Gaze-based Object Detection in the Wild

Daniel Weber, Wolfgang Fuhl, Andreas Zell, Enkelejda Kasneci

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations


I human-robot collaboration, one challenging task is to teach a robot new yet unknown objects enabling it to interact with them. Thereby, gaze can contain valuable information. We investigate if it is possible to detect objects (object or no object) merely from gaze data and determine their bounding box parameters. For this purpose, we explore different sizes of temporal windows, which serve as a basis for the computation of heatmaps, i.e., the spatial distribution of the gaze data. Additionally, we analyze different grid sizes of these heatmaps, and demonstrate the functionality in a proof of concept using different machine learning techniques. Our method is characterized by its speed and resource efficiency compared to conventional object detectors. In order to generate the required data, we conducted a study with five subjects who could move freely and thus, turn towards arbitrary objects. This way, we chose a scenario for our data collection that is as realistic as possible. Since the subjects move while facing objects, the heatmaps also contain gaze data trajectories, complicating the detection and parameter regression. We make our data set publicly available to the research community for download.

Original languageEnglish
Title of host publicationProceedings - 2022 6th IEEE International Conference on Robotic Computing, IRC 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages5
ISBN (Electronic)9781665472609
StatePublished - 2022
Event6th IEEE International Conference on Robotic Computing, IRC 2022 - Virtual, Online, Italy
Duration: 5 Dec 20227 Dec 2022

Publication series

NameProceedings - 2022 6th IEEE International Conference on Robotic Computing, IRC 2022


Conference6th IEEE International Conference on Robotic Computing, IRC 2022
CityVirtual, Online


  • dataset
  • eye tracking
  • gaze
  • heatmap
  • human-robot collaboration
  • machine learning
  • object detection


Dive into the research topics of 'Gaze-based Object Detection in the Wild'. Together they form a unique fingerprint.

Cite this