Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention

Süleyman Özdel, Yao Rong, Berat Mert Albaba, Yen Ling Kuo, Xi Wang, Enkelejda Kasneci

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

1 Zitat (Scopus)

Abstract

Humans utilize their gaze to concentrate on essential information while perceiving and interpreting intentions in videos. Incorporating human gaze into computational algorithms can significantly enhance model performance in video understanding tasks. In this work, we address a challenging and innovative task in video understanding: predicting the actions of an agent in a video based on a partial video. We introduce the Gaze-guided Action Anticipation algorithm, which establishes a visual-semantic graph from the video input. Our method utilizes a Graph Neural Network to recognize the agent's intention and predict the action sequence to fulfill this intention. To assess the efficiency of our approach, we collect a dataset containing household activities generated in the VirtualHome environment, accompanied by human gaze data of viewing videos. Our method outperforms state-of-the-art techniques, achieving a 7% improvement in accuracy for 18-class intention recognition. This highlights the efficiency of our method in learning important features from human gaze data.

OriginalspracheEnglisch
TitelProceedings - ETRA 2024, ACM Symposium on Eye Tracking Research and Applications
Redakteure/-innenStephen N. Spencer
Herausgeber (Verlag)Association for Computing Machinery
ISBN (elektronisch)9798400706073
DOIs
PublikationsstatusVeröffentlicht - 4 Juni 2024
Veranstaltung16th Annual ACM Symposium on Eye Tracking Research and Applications, ETRA 2024 - Hybrid, Glasgow, Großbritannien/Vereinigtes Königreich
Dauer: 4 Juni 20247 Juni 2024

Publikationsreihe

NameEye Tracking Research and Applications Symposium (ETRA)

Konferenz

Konferenz16th Annual ACM Symposium on Eye Tracking Research and Applications, ETRA 2024
Land/GebietGroßbritannien/Vereinigtes Königreich
OrtHybrid, Glasgow
Zeitraum4/06/247/06/24

Fingerprint

Untersuchen Sie die Forschungsthemen von „Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren