Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection

Tim Salzmann, Markus Ryll, Alex Bewley, Matthias Minderer

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

1 Zitat (Scopus)

Abstract

Visual relationship detection aims to identify objects and their relationships in images. Prior methods approach this task by adding separate relationship modules or decoders to existing object detection architectures. This separation increases complexity and hinders end-to-end training, which limits performance. We propose a simple and highly efficient decoder-free architecture for open-vocabulary visual relationship detection. Our model consists of a Transformer-based image encoder that represents objects as tokens and models their relationships implicitly. To extract relationship information, we introduce an attention mechanism that selects object pairs likely to form a relationship. We provide a single-stage recipe to train this model on a mixture of object and relationship detection data. Our approach achieves state-of-the-art relationship detection performance on Visual Genome and on the large-vocabulary GQA benchmark at real-time inference speeds. We provide ablations, real-world qualitative examples, and analyses of zero-shot performance.

OriginalspracheEnglisch
TitelComputer Vision – ECCV 2024 - 18th European Conference, Proceedings
Redakteure/-innenAleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Herausgeber (Verlag)Springer Science and Business Media Deutschland GmbH
Seiten195-213
Seitenumfang19
ISBN (Print)9783031729065
DOIs
PublikationsstatusVeröffentlicht - 2025
Veranstaltung18th European Conference on Computer Vision, ECCV 2024 - Milan, Italien
Dauer: 29 Sept. 20244 Okt. 2024

Publikationsreihe

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Band15142 LNCS
ISSN (Print)0302-9743
ISSN (elektronisch)1611-3349

Konferenz

Konferenz18th European Conference on Computer Vision, ECCV 2024
Land/GebietItalien
OrtMilan
Zeitraum29/09/244/10/24

Fingerprint

Untersuchen Sie die Forschungsthemen von „Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren