Real-time 3D reconstruction for occlusion-aware interactions in mixed reality

Alexander Ladikos, Nassir Navab

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

6 Scopus citations


In this paper, we present a system for performing real-time occlusion-aware interactions in a mixed reality environment. Our system consists of 16 ceiling-mounted cameras observing an interaction space of size 3.70 m x 3.20 m x 2.20 m. We reconstruct the shape of all objects inside the interaction space using a visual hull method at a frame rate of 30 Hz. Due to the interactive speed of the system, the users can act naturally in the interaction space. In addition, since we reconstruct the shape of every object, the users can use their entire body to interact with the virtual objects. This is a significant advantage over marker-based tracking systems, which require a prior setup and tedious calibration steps for every user who wants to use the system. With our system anybody can just enter the interaction space and start interacting naturally. We illustrate the usefulness of our system through two sample applications. The first application is a real-life version of the well known game Pong. With our system, the player can use his whole body as the pad. The second application is concerned with video compositing. It allows a user to integrate himself as well as virtual objects into a prerecorded sequence while correctly handling occlusions.

Original languageEnglish
Title of host publicationAdvances in Visual Computing - 5th International Symposium, ISVC 2009, Proceedings
Number of pages10
EditionPART 1
StatePublished - 2009
Event5th International Symposium on Advances in Visual Computing, ISVC 2009 - Las Vegas, NV, United States
Duration: 30 Nov 20092 Dec 2009

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume5875 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference5th International Symposium on Advances in Visual Computing, ISVC 2009
Country/TerritoryUnited States
CityLas Vegas, NV


Dive into the research topics of 'Real-time 3D reconstruction for occlusion-aware interactions in mixed reality'. Together they form a unique fingerprint.

Cite this