Laparoscopic virtual mirror new interaction paradigm for monitor based augmented reality

A major roadblock for using augmented reality in many medical and industrial applications is the fact that the user cannot take full advantage of the 3D virtual data. This usually requires the user to move the virtual object, which disturbs the real/virtual alignment, or to move his head around the real objects, which is not always possible and/or practical. This problem becomes more dramatic when a single camera is used for monitor based augmentation, such as in augmented laparoscopic surgery. In this paper we introduce an interaction and 3D visualization paradigm, which presents a new solution to this old problem. The interaction paradigm uses an interactive virtual mirror positioned into the augmented scene, which allows easy and complete interactive visualization of 3D virtual data. This paper focuses on the exemplary application of such visualization techniques to laparoscopic interventions. A large number of such interventions aims at regions inside a specific organ, e.g. blood vessels to be clipped for tumor resection. We use high-resolution intra-operative imaging data generated by a mobile C-arm with cone-beam CT imaging capability. Both the C-arm and the laparoscope are optically tracked and registered in a common world coordinate frame. After patient positioning, port placement, and carbon dioxide insufflation, a C-arm volume is reconstructed during patient exhalation and superimposed in real time on the laparoscopic live video without any need for an additional patient registration procedure. To overcome the missing perception of 3D depth and shape when rendering virtual volume data directly on top of the organ's surface view, we introduce the concept of a laparoscopic virtual mirror: A virtual reflection plane within the live laparoscopic video, which is able to visualize a reflected side view of the organ and its interior. This enables the surgeon to observe the 3D structure of, for example, blood vessels by moving the virtual mirror within the augmented monocular view of the laparoscope.