Continuous monitoring of emotions by a multimodal cooperative sensor system

Multimodal emotion recognition is a challenging topic that aims at determining the affective state of a subject by combining audio-visual and physiological signals acquired in a naturalistic environment. This procedure can be used to monitor the emotional state of a subject affected by mental disorder or under medical treatment. Common attempts principally learn a unique complex machine learning system on descriptors collected from different subjects. The novel paradigm of single-subject multimodal regression model (SSMRM) that we propose in this study is embedded in a averaging-based merging strategy that aggregates the responses provided by each model during the test of a new subject. This new approach presents a flexible architecture able to continuously embed new models without global re-training.