Realtime 3D 360-Degree Telepresence with Deep-Learning-Based Head-Motion Prediction

The acceptance and the dissemination of 360° telepresence systems are severely restricted by the appearance of motion sickness. Such systems consist of a client-side, where the user wears a head-mounted display, a server-side, which provides a 3D 360° visual representation of the remote scene, and a communication network in between. Due to the physically unavoidable latency, there is often a noticeable lag between head motion and visual response. If the sensory information from the visual system is not consistent with the perceived ego-motion of the user, and the emergence of visual discomfort is inevitable. In this paper, we present a delay-compensating 3D 360° vision system, which provides omnistereoscopic vision and a significant reduction of the perceived latency. We formally describe the underlying problem and provide an algebraic description of the amount of achievable delay compensation both for perspective and fisheye camera systems, considering all the three degrees of freedom. Furthermore, we propose a generic approach that is agnostic to the underlying camera system. In addition, a novel deep-learning-based head motion prediction algorithm is presented to further improve the compensation rate. Using the naive approach, where no compensation and prediction is applied, we obtain a mean compensation rate of 72.8% for investigated latencies from 0.1 to 1.0 s. Our proposed generic delay-compensation approach, combined with our novel deep-learning-based head-motion prediction approach, manages to achieve a mean compensation rate of 97.3%. The proposed technique also substantially outperforms prior head-motion prediction techniques, both for traditional and deep learning-based methods.