SceneGrok: Inferring action maps in 3D environments

Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner

Research output: Contribution to journalArticlepeer-review

72 Scopus citations


With modern computer graphics, we can generate enormous amounts of 3D scene data. It is now possible to capture highquality 3D representations of large real-world environments. Large shape and scene databases, such as the Trimble 3D Warehouse, are publicly accessible and constantly growing. Unfortunately, while a great amount of 3D content exists, most of it is detached from the semantics and functionality of the objects it represents. In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments. Using RGB-D sensors, we capture dense 3D reconstructions of real-world scenes, and observe and track people as they interact with the environment. With these observations, we train a classifier which can transfer interaction knowledge to unobserved 3D scenes. We predict a likelihood of a given action taking place over all locations in a 3D environment and refer to this representation as an action map over the scene. We demonstrate prediction of action maps in both 3D scans and virtual scenes. We evaluate our predictions against ground truth annotations by people, and present an approach for characterizing 3D scenes by functional similarity using action maps.

Original languageEnglish
JournalACM Transactions on Graphics
Issue number6
StatePublished - 19 Nov 2014
Externally publishedYes


  • 3D scenes
  • Object semantics
  • Scene understanding


Dive into the research topics of 'SceneGrok: Inferring action maps in 3D environments'. Together they form a unique fingerprint.

Cite this