Aggregating physiological and eye tracking signals to predict perception in the absence of ground truth

Enkelejda Kasneci, Thomas Kübler, Klaus Broelemann, Gjergji Kasneci

Research output: Contribution to journalArticlepeer-review

17 Scopus citations


Today's driving assistance systems build on numerous sensors to provide assistance for specific tasks. In order to not patronize the driver, intensity and timing of critical responses by such systems is determined based on parameters derived from vehicle dynamics and scene recognition. However, to date, information on object perception by the driver is not considered by such systems. With advances in eye-tracking technology, a powerful tool to assess the driver's visual perception has become available, which, in many studies, has been integrated with physiological signals, i.e., galvanic skin response and EEG, for reliable prediction of object perception. We address the problem of aggregating binary signals from physiological sensors and eye tracking to predict a driver's visual perception of scene hazards. In the absence of ground truth, it is crucial to use an aggregation scheme that estimates the reliability of each signal source and thus reliably aggregates signals to predict whether an object has been perceived. To this end, we apply state-of-the-art methods for response aggregation on data obtained from simulated driving sessions with 30 subjects. Our results show that a probabilistic aggregation scheme on top of an Expectation-Maximization-based estimation of source reliabilities can predict hazard perception at a recall and precision of 96% in real-time.

Original languageEnglish
Pages (from-to)450-455
Number of pages6
JournalComputers in Human Behavior
StatePublished - 1 Mar 2017
Externally publishedYes


  • ECG
  • Eye tracking
  • Eye tracking
  • Galvanic skin response
  • Hazard perception
  • Physiological signals
  • Pupil dilation
  • Response aggregation


Dive into the research topics of 'Aggregating physiological and eye tracking signals to predict perception in the absence of ground truth'. Together they form a unique fingerprint.

Cite this