SupeRGB-D: Zero-Shot Instance Segmentation in Cluttered Indoor Environments

Evin Pnar Ornek, Aravindhan K. Krishnan, Shreekant Gayaka, Cheng Hao Kuo, Arnie Sen, Nassir Navab, Federico Tombari

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Object instance segmentation is a key challenge for indoor robots navigating cluttered environments with many small objects. Limitations in 3D sensing capabilities often make it difficult to detect every possible object. While deep learning approaches may be effective for this problem, manually annotating 3D data for supervised learning is time-consuming. In this work, we explore zero-shot instance segmentation (ZSIS) from RGB-D data to identify unseen objects in a semantic category-agnostic manner. We introduce a zero-shot split for Tabletop Objects Dataset (TOD-Z) to enable this study and present a method that uses annotated objects to learn the 'objectness' of pixels and generalize to unseen object categories in cluttered indoor environments. Our method, SupeRGB-D, groups pixels into small patches based on geometric cues and learns to merge the patches in a deep agglomerative clustering fashion. SupeRGB-D outperforms existing baselines on unseen objects while achieving similar performance on seen objects. We further show competitive results on the real dataset OCID. With its lightweight design (0.4 MB memory requirement), our method is extremely suitable for mobile and robotic applications. Additional DINO features can increase the performance with a higher memory requirement.

Original languageEnglish
Pages (from-to)3709-3716
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume8
Issue number6
DOIs
StatePublished - 1 Jun 2023

Keywords

  • RGB-D Perception
  • deep Learning for visual perception
  • object detection
  • segmentation and categorization

Fingerprint

Dive into the research topics of 'SupeRGB-D: Zero-Shot Instance Segmentation in Cluttered Indoor Environments'. Together they form a unique fingerprint.

Cite this