SemanticPaint: Interactive segmentation and learning of 3D world

Julien Valentin, Pushmeet Kohli, Vibhav Vineet, Matthias Nießner, Ming Ming Cheng, Antonio Criminisi, David Kim, Shahram Izadi, Jamie Shotton, Philip Torr

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Scopus citations

Abstract

We present a real-time, interactive system for the geometric reconstruction, object-class segmentation and learning of 3D scenes [Valentin et al.]. Using our system, a user can walk into a room wearing a consumer depth camera and a virtual reality headset, and both densely reconstruct the 3D scene [Niessner et al. 2013]) and interactively segment the environment into object classes such as 'chair', 'floor' and 'table'. The user interacts physically with the real-world scene, touching or pointing at objects and using voice commands to assign them appropriate labels. These user generated labels are leveraged by a new online random forest-based machine learning algorithm, which is used to predict labels for previously unseen parts of the scene. The predicted labels, together with those provided directly by the user, are incorporated into a dense 3D conditional random field model, over which we perform mean-field inference to filter out label inconsistencies. The entire pipeline runs in real time, and the user stays 'in the loop' throughout the process, receiving immediate feedback about the progress of the labelling and interacting with the scene as necessary to refine the predicted segmentation.

Original languageEnglish
Title of host publicationACM SIGGRAPH 2015 Talks, SIGGRAPH 2015
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450336369
DOIs
StatePublished - 31 Jul 2015
Externally publishedYes
EventInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015 - Los Angeles, United States
Duration: 9 Aug 201513 Aug 2015

Publication series

NameACM SIGGRAPH 2015 Talks, SIGGRAPH 2015

Conference

ConferenceInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015
Country/TerritoryUnited States
CityLos Angeles
Period9/08/1513/08/15

Fingerprint

Dive into the research topics of 'SemanticPaint: Interactive segmentation and learning of 3D world'. Together they form a unique fingerprint.

Cite this