A systems theoretical model for human perception in multimodal presence systems

Advanced virtual environment and telepresence systems, here termed presence systems, are to display a continuous flow of information to the human operator in several modalities of human perception. Such multimodal stimulation enhances the quality of the human operator's interaction with the remote, or virtual environment. Dynamics and bilaterality in interaction between operator and environment complicate model-based display to the operator, particularly for the human proximity (haptic) senses. Since the human operator then is a part of the closed-control loop, research on multimodal and haptic feedback combines mechatronics engineering challenges with human factors. In this paper, it is shown how a mathematical model of the human perception process, in the terminology of systems theory, can contribute to convergence and progress in an interdisciplinary research field with regard to problem structuring, systematic planning and evaluation of experiments, and model-based generation of multimodal stimuli. The proposed systems theoretical framework describes the principles of human perception as a concatenation of nonlinear vector mappings. The efficacy and benefits of the presented theory are demonstrated by formal descriptions of sensory substitution experimental results found in the literature, and novel experiments performed in a virtual environment.