Fusion of multi-modal sensors in a voxel occupancy grid for tracking and behaviour analysis

In this paper, we present a multi-modal fusion scheme for tracking and behavior analysis in Smart Home environments. This is applied to tracking multiple people and detecting their behavior. To this end, information from multiple heterogeneous sensors (visual color sensor, thermal sensor, infrared sensor and photonic mixer devices) is combined in a common 3D voxel occupancy grid. Graph cuts are used for data fusion and to accurately reconstruct people in the scene. A Viterbi tracking framework is applied to track all people and simultaneously determine their behaviour. We evaluate the proposed fusion scheme on the PROMETHEUS Smart Home database and show the impact of different sensors and modalities to the final results.