Skip to content
Show report in:

UMINF 09.09

The brain and interaction in a multimodal reality

This paper reviews the neural correlates of multimodal integration and the role it plays in the creation and maintenance of perception of reality. These issues are illuminated by reviewing concepts and research from a number of related subjects and we explore some of the relevant cognitive models, such as the memory-prediction framework. We further focus on how multimodal integration affects reality-based interaction (RBI) in general and virtual reality (VR) in particular. In this case the reality in question is generated by a computer and perception of reality may be unstable. In VR-related research the quality of the perception of reality is commonly referred to as presence and a review of the conditions for and effects of varying degrees of presence is presented. An increased understanding of the role of multimodal integration in the creation and maintenance of presence is one of the primary goals of this paper. The hope is that this will help us to understand and improve presence, something that we will show to be of great value. The effect of disturbances and failure in the multimodal integration on the perception of reality and presence is of particular interest. This is related to the concept of breaks in presence and prediction errors, to provide some framework for understanding. Also, the importance of understanding the neural correlates of these cognitive functions is related to the possible use of VR in combination with brain imaging, exemplified with functional magnetic resonance imaging (fMRI). Finally, we discuss possible future work and possibilities to advance the understanding of the brain and reality in the context of human computer interaction.


interaction, brain, VR, fMRI, multimodal, reality, reality-based interaction


Back Edit this report
Entry responsible: Account Deleted - might not work

Page Responsible: Frank Drewes