A Personalised Approach to Audiovisual Humour Recognition and its Individual-level Fairness

Alexander Kathan, Shahin Amiriparian, Lukas Christ, Andreas Triantafyllopoulos, Niklas Müller, Andreas König, Björn W. Schuller

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

8 Scopus citations

Abstract

Humour is one of the most subtle and contextualised behavioural patterns to study in social psychology and has a major impact on human emotions, social cognition, behaviour, and relations. Consequently, an automatic understanding of humour is crucial and challenging for a naturalistic human-robot interaction. Recent artificial intelligence (AI)-based methods have shown progress in multimodal humour recognition. However, such methods lack a mechanism in adapting to each individual's characteristics, resulting in a decreased performance, e.g., due to different facial expressions. Further, these models are faced with generalisation problems when being applied for recognition of different styles of humour. We aim to address these challenges by introducing a novel multimodal humour recognition approach in which the models are personalised for each individual in the Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset. We begin by training a model on all individuals in the dataset. Subsequently, we fine-tune all layers of this model with the data from each individual. Finally, we use these models for the prediction task. Using the proposed personalised models, it is possible to significantly (two-tailed t-test, p < 0.05) outperform the non-personalised models. In particular, the mean Area Under the Curve (AUC) is increased from.7573 to.7731 for the audio modality, and from.9203 to.9256 for the video modality. In addition, we apply a weighted late fusion approach which increases the overall performance to an AUC of.9308, demonstrating the complementarity of the features. Finally, we evaluate the individual-level fairness of our approach and show which group of subjects benefits most of using personalisation.

Original languageEnglish
Title of host publicationMuSe 2022 - Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge
PublisherAssociation for Computing Machinery, Inc
Pages29-36
Number of pages8
ISBN (Electronic)9781450394840
DOIs
StatePublished - 10 Oct 2022
Externally publishedYes
Event3rd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2022, held in conjunction with the ACM Multimedia 2022 - Lisboa, Portugal
Duration: 10 Oct 2022 → …

Publication series

NameMuSe 2022 - Proceedings of the 3rd International Multimodal Sentiment Analysis Workshop and Challenge

Conference

Conference3rd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2022, held in conjunction with the ACM Multimedia 2022
Country/TerritoryPortugal
CityLisboa
Period10/10/22 → …

Keywords

  • automatic humour recognition
  • personalised models
  • speech processing
  • video processing

Fingerprint

Dive into the research topics of 'A Personalised Approach to Audiovisual Humour Recognition and its Individual-level Fairness'. Together they form a unique fingerprint.

Cite this