Holistic OR domain modeling: a semantic scene graph approach

Ege Özsoy, Tobias Czempiel, Evin Pınar Örnek, Ulrich Eck, Federico Tombari, Nassir Navab

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Purpose: Surgical procedures take place in highly complex operating rooms (OR), involving medical staff, patients, devices and their interactions. Until now, only medical professionals are capable of comprehending these intricate links and interactions. This work advances the field toward automated, comprehensive and semantic understanding and modeling of the OR domain by introducing semantic scene graphs (SSG) as a novel approach to describing and summarizing surgical environments in a structured and semantically rich manner. Methods: We create the first open-source 4D SSG dataset. 4D-OR includes simulated total knee replacement surgeries captured by RGB-D sensors in a realistic OR simulation center. It includes annotations for SSGs, human and object pose, clinical roles and surgical phase labels. We introduce a neural network-based SSG generation pipeline for semantic reasoning in the OR and apply our approach to two downstream tasks: clinical role prediction and surgical phase recognition. Results: We show that our pipeline can successfully reason within the OR domain. The capabilities of our scene graphs are further highlighted by their successful application to clinical role prediction and surgical phase recognition tasks. Conclusion: This work paves the way for multimodal holistic operating room modeling, with the potential to significantly enhance the state of the art in surgical data analysis, such as enabling more efficient and precise decision-making during surgical procedures, and ultimately improving patient safety and surgical outcomes. We release our code and dataset at github.com/egeozsoy/4D-OR.

Original languageEnglish
Pages (from-to)791-799
Number of pages9
JournalInternational Journal of Computer Assisted Radiology and Surgery
Volume19
Issue number5
DOIs
StatePublished - May 2024

Keywords

  • 3D
  • 4D-OR
  • Semantic scene graph
  • Surgical scene understanding

Fingerprint

Dive into the research topics of 'Holistic OR domain modeling: a semantic scene graph approach'. Together they form a unique fingerprint.

Cite this