Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks

Florian Eyben; Stavros Petridis; Björn Schuller; George Tzimiropoulos; Stefanos Zafeiriou; Maja Pantic

doi:10.1109/ICASSP.2011.5947690

Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks

Florian Eyben, Stavros Petridis, Björn Schuller, George Tzimiropoulos, Stefanos Zafeiriou, Maja Pantic

Chair of Health Informatics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

21 Scopus citations

Abstract

We investigate classification of non-linguistic vocalisations with a novel audiovisual approach and Long Short-Term Memory (LSTM) Recurrent Neural Networks as highly successful dynamic sequence classifiers. As database of evaluation serves this year's Paralinguistic Challenge's Audiovisual Interest Corpus of human-to-human natural conversation. For video-based analysis we compare shape and appearance based features. These are fused in an early manner with typical audio descriptors. The results show significant improvements of LSTM networks over a static approach based on Support Vector Machines. More important, we can show a significant gain in performance when fusing audio and visual shape features.

Original language	English
Title of host publication	2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings
Pages	5844-5847
Number of pages	4
DOIs	https://doi.org/10.1109/ICASSP.2011.5947690
State	Published - 2011
Event	36th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Prague, Czech Republic Duration: 22 May 2011 → 27 May 2011

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
ISSN (Print)	1520-6149

Conference

Conference	36th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011
Country/Territory	Czech Republic
City	Prague
Period	22/05/11 → 27/05/11

Keywords

Audio-visual Processing
Laughter
Long Short-Term Memory
Non-linguistic Vocalisations

Access to Document

10.1109/ICASSP.2011.5947690

Cite this

Eyben, F., Petridis, S., Schuller, B., Tzimiropoulos, G., Zafeiriou, S., & Pantic, M. (2011). Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks. In 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings (pp. 5844-5847). Article 5947690 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). https://doi.org/10.1109/ICASSP.2011.5947690

Eyben, Florian ; Petridis, Stavros ; Schuller, Björn et al. / Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks. 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings. 2011. pp. 5844-5847 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

@inproceedings{1dc5d12ba5a546cfbfe121eab4a12e45,

title = "Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks",

abstract = "We investigate classification of non-linguistic vocalisations with a novel audiovisual approach and Long Short-Term Memory (LSTM) Recurrent Neural Networks as highly successful dynamic sequence classifiers. As database of evaluation serves this year's Paralinguistic Challenge's Audiovisual Interest Corpus of human-to-human natural conversation. For video-based analysis we compare shape and appearance based features. These are fused in an early manner with typical audio descriptors. The results show significant improvements of LSTM networks over a static approach based on Support Vector Machines. More important, we can show a significant gain in performance when fusing audio and visual shape features.",

keywords = "Audio-visual Processing, Laughter, Long Short-Term Memory, Non-linguistic Vocalisations",

author = "Florian Eyben and Stavros Petridis and Bj{\"o}rn Schuller and George Tzimiropoulos and Stefanos Zafeiriou and Maja Pantic",

year = "2011",

doi = "10.1109/ICASSP.2011.5947690",

language = "English",

isbn = "9781457705397",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

pages = "5844--5847",

booktitle = "2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings",

note = "36th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 ; Conference date: 22-05-2011 Through 27-05-2011",

}

Eyben, F, Petridis, S, Schuller, B, Tzimiropoulos, G, Zafeiriou, S & Pantic, M 2011, Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks. in 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings., 5947690, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, pp. 5844-5847, 36th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011, Prague, Czech Republic, 22/05/11. https://doi.org/10.1109/ICASSP.2011.5947690

Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks. / Eyben, Florian; Petridis, Stavros; Schuller, Björn et al.
2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings. 2011. p. 5844-5847 5947690 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks

AU - Eyben, Florian

AU - Petridis, Stavros

AU - Schuller, Björn

AU - Tzimiropoulos, George

AU - Zafeiriou, Stefanos

AU - Pantic, Maja

PY - 2011

Y1 - 2011

N2 - We investigate classification of non-linguistic vocalisations with a novel audiovisual approach and Long Short-Term Memory (LSTM) Recurrent Neural Networks as highly successful dynamic sequence classifiers. As database of evaluation serves this year's Paralinguistic Challenge's Audiovisual Interest Corpus of human-to-human natural conversation. For video-based analysis we compare shape and appearance based features. These are fused in an early manner with typical audio descriptors. The results show significant improvements of LSTM networks over a static approach based on Support Vector Machines. More important, we can show a significant gain in performance when fusing audio and visual shape features.

AB - We investigate classification of non-linguistic vocalisations with a novel audiovisual approach and Long Short-Term Memory (LSTM) Recurrent Neural Networks as highly successful dynamic sequence classifiers. As database of evaluation serves this year's Paralinguistic Challenge's Audiovisual Interest Corpus of human-to-human natural conversation. For video-based analysis we compare shape and appearance based features. These are fused in an early manner with typical audio descriptors. The results show significant improvements of LSTM networks over a static approach based on Support Vector Machines. More important, we can show a significant gain in performance when fusing audio and visual shape features.

KW - Audio-visual Processing

KW - Laughter

KW - Long Short-Term Memory

KW - Non-linguistic Vocalisations

UR - http://www.scopus.com/inward/record.url?scp=80051631670&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2011.5947690

DO - 10.1109/ICASSP.2011.5947690

M3 - Conference contribution

AN - SCOPUS:80051631670

SN - 9781457705397

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 5844

EP - 5847

BT - 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings

T2 - 36th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011

Y2 - 22 May 2011 through 27 May 2011

ER -

Eyben F, Petridis S, Schuller B, Tzimiropoulos G, Zafeiriou S, Pantic M. Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks. In 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011 - Proceedings. 2011. p. 5844-5847. 5947690. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). doi: 10.1109/ICASSP.2011.5947690

Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this