MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox

Lukas Stappen; Lea Schumann; Benjamin Sertolli; Alice Baird; Benjamin Weigell; Erik Cambria; Björn W. Schuller

doi:10.1145/3475957.3484451

MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox

Lukas Stappen, Lea Schumann, Benjamin Sertolli, Alice Baird, Benjamin Weigell, Erik Cambria, Björn W. Schuller

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

43 Scopus citations

Abstract

We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation is out-of-the-box available with all dependencies using a Docker container.

Original language	English
Title of host publication	MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021
Publisher	Association for Computing Machinery, Inc
Pages	75-82
Number of pages	8
ISBN (Electronic)	9781450386784
DOIs	https://doi.org/10.1145/3475957.3484451
State	Published - 24 Oct 2021
Externally published	Yes
Event	2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021 - Virtual, Online, China Duration: 24 Oct 2021 → 24 Oct 2021

Publication series

Name	MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021

Conference

Conference	2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021
Country/Territory	China
City	Virtual, Online
Period	24/10/21 → 24/10/21

Keywords

affective computing
annotation
emotion classes
emotion recognition
gold-standard
multimodal sentiment analysis
smoothing

Access to Document

10.1145/3475957.3484451

Cite this

Stappen, L., Schumann, L., Sertolli, B., Baird, A., Weigell, B., Cambria, E., & Schuller, B. W. (2021). MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. In MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021 (pp. 75-82). (MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021). Association for Computing Machinery, Inc. https://doi.org/10.1145/3475957.3484451

Stappen, Lukas ; Schumann, Lea ; Sertolli, Benjamin et al. / MuSe-Toolbox : The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021. Association for Computing Machinery, Inc, 2021. pp. 75-82 (MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021).

@inproceedings{0645f6c3992b4e908cd28f80a9cae78f,

title = "MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox",

abstract = "We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation is out-of-the-box available with all dependencies using a Docker container.",

keywords = "affective computing, annotation, emotion classes, emotion recognition, gold-standard, multimodal sentiment analysis, smoothing",

author = "Lukas Stappen and Lea Schumann and Benjamin Sertolli and Alice Baird and Benjamin Weigell and Erik Cambria and Schuller, {Bj{\"o}rn W.}",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021 ; Conference date: 24-10-2021 Through 24-10-2021",

year = "2021",

month = oct,

day = "24",

doi = "10.1145/3475957.3484451",

language = "English",

series = "MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021",

publisher = "Association for Computing Machinery, Inc",

pages = "75--82",

booktitle = "MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021",

}

Stappen, L, Schumann, L, Sertolli, B, Baird, A, Weigell, B, Cambria, E & Schuller, BW 2021, MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. in MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021. MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021, Association for Computing Machinery, Inc, pp. 75-82, 2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021, Virtual, Online, China, 24/10/21. https://doi.org/10.1145/3475957.3484451

MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. / Stappen, Lukas; Schumann, Lea; Sertolli, Benjamin et al.
MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021. Association for Computing Machinery, Inc, 2021. p. 75-82 (MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MuSe-Toolbox

T2 - 2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021

AU - Stappen, Lukas

AU - Schumann, Lea

AU - Sertolli, Benjamin

AU - Baird, Alice

AU - Weigell, Benjamin

AU - Cambria, Erik

AU - Schuller, Björn W.

PY - 2021/10/24

Y1 - 2021/10/24

N2 - We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation is out-of-the-box available with all dependencies using a Docker container.

AB - We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation is out-of-the-box available with all dependencies using a Docker container.

KW - affective computing

KW - annotation

KW - emotion classes

KW - emotion recognition

KW - gold-standard

KW - multimodal sentiment analysis

KW - smoothing

UR - http://www.scopus.com/inward/record.url?scp=85118977450&partnerID=8YFLogxK

U2 - 10.1145/3475957.3484451

DO - 10.1145/3475957.3484451

M3 - Conference contribution

AN - SCOPUS:85118977450

T3 - MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021

SP - 75

EP - 82

BT - MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021

PB - Association for Computing Machinery, Inc

Y2 - 24 October 2021 through 24 October 2021

ER -

Stappen L, Schumann L, Sertolli B, Baird A, Weigell B, Cambria E et al. MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. In MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021. Association for Computing Machinery, Inc. 2021. p. 75-82. (MuSe 2021 - Proceedings of the 2nd Multimodal Sentiment Analysis Challenge, co-located with ACM MM 2021). doi: 10.1145/3475957.3484451

MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this