Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality

A lack of telepresence is one of the main challenges of vehicle teleoperation, which degrades the task performance of the human operator. This paper introduces a novel human-machine interface (HMI) using a head-mounted display (HMD) to improve the situation awareness and, consequently, the telepresence of the human operator. The proposed HMI concept uses the transmitted data of the camera as well as LiDAR sensors of the remote vehicle to illustrate the 360° vehicle surroundings as a mixture of the real and virtual environments to the human operator. The resulting system provides the possibility to precisely control the remote vehicle with a low additional load to the transmitted data. The developed concept is evaluated by experienced operators within different test scenarios. The results of the test drives show a significant improvement of the task performance as well as a reduction of the workload of the human operators using the proposed HMI concept during control of the teleoperated vehicle.