A Personalised Approach to Audiovisual Humour Recognition and its Individual-level Fairness

Alexander Kathan, Shahin Amiriparian, Lukas Christ, Andreas Triantafyllopoulos, Niklas Müller, Andreas König, Björn W. Schuller

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

15 Zitate (Scopus)

Abstract

Humour is one of the most subtle and contextualised behavioural patterns to study in social psychology and has a major impact on human emotions, social cognition, behaviour, and relations. Consequently, an automatic understanding of humour is crucial and challenging for a naturalistic human-robot interaction. Recent artificial intelligence (AI)-based methods have shown progress in multimodal humour recognition. However, such methods lack a mechanism in adapting to each individual's characteristics, resulting in a decreased performance, e.g., due to different facial expressions. Further, these models are faced with generalisation problems when being applied for recognition of different styles of humour. We aim to address these challenges by introducing a novel multimodal humour recognition approach in which the models are personalised for each individual in the Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset. We begin by training a model on all individuals in the dataset. Subsequently, we fine-tune all layers of this model with the data from each individual. Finally, we use these models for the prediction task. Using the proposed personalised models, it is possible to significantly (two-tailed t-test, p < 0.05) outperform the non-personalised models. In particular, the mean Area Under the Curve (AUC) is increased from.7573 to.7731 for the audio modality, and from.9203 to.9256 for the video modality. In addition, we apply a weighted late fusion approach which increases the overall performance to an AUC of.9308, demonstrating the complementarity of the features. Finally, we evaluate the individual-level fairness of our approach and show which group of subjects benefits most of using personalisation.

OriginalspracheEnglisch
TitelMuSe 2022 - Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge
Herausgeber (Verlag)Association for Computing Machinery, Inc
Seiten29-36
Seitenumfang8
ISBN (elektronisch)9781450394840
DOIs
PublikationsstatusVeröffentlicht - 10 Okt. 2022
Extern publiziertJa
Veranstaltung3rd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2022, held in conjunction with the ACM Multimedia 2022 - Lisboa, Portugal
Dauer: 10 Okt. 2022 → …

Publikationsreihe

NameMuSe 2022 - Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge

Konferenz

Konferenz3rd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2022, held in conjunction with the ACM Multimedia 2022
Land/GebietPortugal
OrtLisboa
Zeitraum10/10/22 → …

Fingerprint

Untersuchen Sie die Forschungsthemen von „A Personalised Approach to Audiovisual Humour Recognition and its Individual-level Fairness“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren