Development of a multi-modal multi-user telepresence and teleaction system

The presented multi-user telepresence and teleaction system enables two teleoperators, which are independently controlled by two human operators, to perform collaborative actions in a remote environment. The system provides the operators with visual, auditory, and haptic feedback and allows them to physically interact with objects in the remote environment. The interaction between the two human operators is enhanced by augmenting visual and auditory feedback. The paper is focused on the amalgamation of the individual subsystems, which handle the different modalities, into one tightly integrated system, which creates a common workspace for two operators. The overall system architecture and the appropriate design of the auditory, visual, and haptic subsystems are discussed. The audio system makes use of a novel high-fidelity interpolation technique to render three-dimensional sound scenes for both human operators, which enhances their interaction. The design of the video system allows modeling and rendering of the remote environment in real time while regarding changes in the scene by perpetually updating the model. The haptic system is based on admittance-type devices, which are best fitted for applications involving large workspaces and high interaction forces. In addition, the implementation of locomotion techniques for telepresence in large-scale environments are presented. Finally, an application example shows that the system can be successfully employed in a remote maintenance task, which consists of exploring a large-scale environment, moving to the target area, and finally repairing a broken pipe by attaching a sealing clamp. The example demonstrates the necessity of multi-user telepresence and teleaction systems and supports the benefits of consistent multi-modal feedback.