Measuring auditory attention effort in virtual audio-visual environments using pupillometry

Pupillometry is a common approach to measure cognitive effort due to attending auditory targets in noise. The enhanced attention to sound sources required in difficult listening situations is associated with cognitive effort, which is linked with an increased pupil diameter. However, pupil diameter is far more affected by the luminance of the focused visual object. Hence, pupillometry is done with low illumination or by focusing on a region with constant luminance. In ecologically valid audio-visual environments, visual objects appear interactively, dynamically in view and luminance changes constantly. We present a method that compensates for these 'environmental luminance effects' such that the measurement of attention through pupil dilation is possible. Using a head-mounted world-camera and an eye-tracker, luminance in the focused area is measured. A system of differential equations is solved to dynamically predict a baseline pupil diameter as a function of momentary luminance. The approach is verified in a listening experiment with cued attention to one of two spatialized talkers and varying difficulty, while luminance on a video projection screen is varied. Results show that the large effect of luminance on pupil diameter can predicted with sufficient accuracy in order to measure the residual small effect of attentional effort. This paves the way for dynamic attention measurements in virtual communication scenarios.