Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

Fabien Ringeval, Florian Eyben, Eleni Kroupi, Anil Yuce, Jean Philippe Thiran, Touradj Ebrahimi, Denis Lalanne, Björn Schuller

Research output: Contribution to journalArticlepeer-review

158 Scopus citations


Automatic emotion recognition systems based on supervised machine learning require reliable annotation of affective behaviours to build useful models. Whereas the dimensional approach is getting more and more popular for rating affective behaviours in continuous time domains, e.g., arousal and valence, methodologies to take into account reaction lags of the human raters are still rare. We therefore investigate the relevance of using machine learning algorithms able to integrate contextual information in the modelling, like long short-term memory recurrent neural networks do, to automatically predict emotion from several (asynchronous) raters in continuous time domains, i.e., arousal and valence. Evaluations are performed on the recently proposed RECOLA multimodal database (27 subjects, 5 min of data and six raters for each), which includes audio, video, and physiological (ECG, EDA) data. In fact, studies uniting audiovisual and physiological information are still very rare. Features are extracted with various window sizes for each modality and performance for the automatic emotion prediction is compared for both different architectures of neural networks and fusion approaches (feature-level/decision-level). The results show that: (i) LSTM network can deal with (asynchronous) dependencies found between continuous ratings of emotion with video data, (ii) the prediction of the emotional valence requires longer analysis window than for arousal and (iii) a decision-level fusion leads to better performance than a feature-level fusion. The best performance (concordance correlation coefficient) for the multimodal emotion prediction is 0.804 for arousal and 0.528 for valence.

Original languageEnglish
Pages (from-to)22-30
Number of pages9
JournalPattern Recognition Letters
StatePublished - 15 Nov 2015


  • Audiovisual and physiological data
  • Context-learning long short-term memory recurrent neural networks
  • Continuous affect analysis
  • Multi-task learning
  • Multimodal fusion
  • Multitime resolution features extraction


Dive into the research topics of 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data'. Together they form a unique fingerprint.

Cite this