Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

Meishu Song; Zijiang Yang; Andreas Triantafyllopoulos; Toru Nakamura; Yongxin Zhang; Zhao Ren; Hiroki Takeuchi; Akifumi Kishi; Tetsuro Ishizawa; Kazuhiro Yoshiuchi; Haojie Zhang; Kun Qian; Bin Hu; Bjorn W. Schuller; Yoshiharu Yamamoto

doi:10.1109/ICDMW60847.2023.00167

Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

Meishu Song
, Zijiang Yang
, Andreas Triantafyllopoulos
, Toru Nakamura
, Yongxin Zhang
, Zhao Ren
, Hiroki Takeuchi
, Akifumi Kishi
, Tetsuro Ishizawa
, Kazuhiro Yoshiuchi
, Haojie Zhang
, Kun Qian
, Bin Hu
, Bjorn W. Schuller
, Yoshiharu Yamamoto

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

The rapid advancement of wearable sensors and machine learning technologies has opened new avenues for mental health monitoring. Despite these advancements, conventional approaches often fail to provide an accurate and personalised understanding of an individual's multi-dimensional emotional state. This paper introduces a novel approach for enhanced daily mental health prediction, focusing on nine distinct emotional states. Our method employs a personalised crossmodal transformer architecture that effectively integrates ZCM (Zero Crossing Mode) and PIM (Proportional Integration Mode) physical signals obtained from piezoelectric accelerometers worn on the non-dominant wrist. Utilising this personalised crossmodal transformer model, our approach adaptively focuses on the most pertinent features across these diverse physical signals, thereby offering a more nuanced and individualised assessment of an individual's emotional state. Our experiments show a considerable improvement in performance, achieving a Concordance Correlation Coefficient (CCC) of 0.475 over a baseline of 0.281.

Original language	English
Title of host publication	Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023
Editors	Jihe Wang, Yi He, Thang N. Dinh, Christan Grant, Meikang Qiu, Witold Pedrycz
Publisher	IEEE Computer Society
Pages	1299-1305
Number of pages	7
ISBN (Electronic)	9798350381641
DOIs	https://doi.org/10.1109/ICDMW60847.2023.00167
State	Published - 2023
Externally published	Yes
Event	23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 - Shanghai, China Duration: 1 Dec 2023 → 4 Dec 2023

Publication series

Name	IEEE International Conference on Data Mining Workshops, ICDMW
ISSN (Print)	2375-9232
ISSN (Electronic)	2375-9259

Conference

Conference	23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023
Country/Territory	China
City	Shanghai
Period	1/12/23 → 4/12/23

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Crossmodal
Mental Health
Personalisation
Physical Signals
Transformer

Access to Document

10.1109/ICDMW60847.2023.00167

Cite this

Song, M., Yang, Z., Triantafyllopoulos, A., Nakamura, T., Zhang, Y., Ren, Z., Takeuchi, H., Kishi, A., Ishizawa, T., Yoshiuchi, K., Zhang, H., Qian, K., Hu, B., Schuller, B. W., & Yamamoto, Y. (2023). Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction. In J. Wang, Y. He, T. N. Dinh, C. Grant, M. Qiu, & W. Pedrycz (Eds.), Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 (pp. 1299-1305). (IEEE International Conference on Data Mining Workshops, ICDMW). IEEE Computer Society. https://doi.org/10.1109/ICDMW60847.2023.00167

Song, Meishu ; Yang, Zijiang ; Triantafyllopoulos, Andreas et al. / Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction. Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. editor / Jihe Wang ; Yi He ; Thang N. Dinh ; Christan Grant ; Meikang Qiu ; Witold Pedrycz. IEEE Computer Society, 2023. pp. 1299-1305 (IEEE International Conference on Data Mining Workshops, ICDMW).

@inproceedings{a37e6b2709484b80abfa3f8e6693eb77,

title = "Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction",

abstract = "The rapid advancement of wearable sensors and machine learning technologies has opened new avenues for mental health monitoring. Despite these advancements, conventional approaches often fail to provide an accurate and personalised understanding of an individual's multi-dimensional emotional state. This paper introduces a novel approach for enhanced daily mental health prediction, focusing on nine distinct emotional states. Our method employs a personalised crossmodal transformer architecture that effectively integrates ZCM (Zero Crossing Mode) and PIM (Proportional Integration Mode) physical signals obtained from piezoelectric accelerometers worn on the non-dominant wrist. Utilising this personalised crossmodal transformer model, our approach adaptively focuses on the most pertinent features across these diverse physical signals, thereby offering a more nuanced and individualised assessment of an individual's emotional state. Our experiments show a considerable improvement in performance, achieving a Concordance Correlation Coefficient (CCC) of 0.475 over a baseline of 0.281.",

keywords = "Crossmodal, Mental Health, Personalisation, Physical Signals, Transformer",

author = "Meishu Song and Zijiang Yang and Andreas Triantafyllopoulos and Toru Nakamura and Yongxin Zhang and Zhao Ren and Hiroki Takeuchi and Akifumi Kishi and Tetsuro Ishizawa and Kazuhiro Yoshiuchi and Haojie Zhang and Kun Qian and Bin Hu and Schuller, \{Bjorn W.\} and Yoshiharu Yamamoto",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 ; Conference date: 01-12-2023 Through 04-12-2023",

year = "2023",

doi = "10.1109/ICDMW60847.2023.00167",

language = "English",

series = "IEEE International Conference on Data Mining Workshops, ICDMW",

publisher = "IEEE Computer Society",

pages = "1299--1305",

editor = "Jihe Wang and Yi He and Dinh, \{Thang N.\} and Christan Grant and Meikang Qiu and Witold Pedrycz",

booktitle = "Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023",

}

Song, M, Yang, Z, Triantafyllopoulos, A, Nakamura, T, Zhang, Y, Ren, Z, Takeuchi, H, Kishi, A, Ishizawa, T, Yoshiuchi, K, Zhang, H, Qian, K, Hu, B, Schuller, BW & Yamamoto, Y 2023, Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction. in J Wang, Y He, TN Dinh, C Grant, M Qiu & W Pedrycz (eds), Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. IEEE International Conference on Data Mining Workshops, ICDMW, IEEE Computer Society, pp. 1299-1305, 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023, Shanghai, China, 1/12/23. https://doi.org/10.1109/ICDMW60847.2023.00167

Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction. / Song, Meishu; Yang, Zijiang; Triantafyllopoulos, Andreas et al.
Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. ed. / Jihe Wang; Yi He; Thang N. Dinh; Christan Grant; Meikang Qiu; Witold Pedrycz. IEEE Computer Society, 2023. p. 1299-1305 (IEEE International Conference on Data Mining Workshops, ICDMW).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

AU - Song, Meishu

AU - Yang, Zijiang

AU - Triantafyllopoulos, Andreas

AU - Nakamura, Toru

AU - Zhang, Yongxin

AU - Ren, Zhao

AU - Takeuchi, Hiroki

AU - Kishi, Akifumi

AU - Ishizawa, Tetsuro

AU - Yoshiuchi, Kazuhiro

AU - Zhang, Haojie

AU - Qian, Kun

AU - Hu, Bin

AU - Schuller, Bjorn W.

AU - Yamamoto, Yoshiharu

PY - 2023

Y1 - 2023

N2 - The rapid advancement of wearable sensors and machine learning technologies has opened new avenues for mental health monitoring. Despite these advancements, conventional approaches often fail to provide an accurate and personalised understanding of an individual's multi-dimensional emotional state. This paper introduces a novel approach for enhanced daily mental health prediction, focusing on nine distinct emotional states. Our method employs a personalised crossmodal transformer architecture that effectively integrates ZCM (Zero Crossing Mode) and PIM (Proportional Integration Mode) physical signals obtained from piezoelectric accelerometers worn on the non-dominant wrist. Utilising this personalised crossmodal transformer model, our approach adaptively focuses on the most pertinent features across these diverse physical signals, thereby offering a more nuanced and individualised assessment of an individual's emotional state. Our experiments show a considerable improvement in performance, achieving a Concordance Correlation Coefficient (CCC) of 0.475 over a baseline of 0.281.

AB - The rapid advancement of wearable sensors and machine learning technologies has opened new avenues for mental health monitoring. Despite these advancements, conventional approaches often fail to provide an accurate and personalised understanding of an individual's multi-dimensional emotional state. This paper introduces a novel approach for enhanced daily mental health prediction, focusing on nine distinct emotional states. Our method employs a personalised crossmodal transformer architecture that effectively integrates ZCM (Zero Crossing Mode) and PIM (Proportional Integration Mode) physical signals obtained from piezoelectric accelerometers worn on the non-dominant wrist. Utilising this personalised crossmodal transformer model, our approach adaptively focuses on the most pertinent features across these diverse physical signals, thereby offering a more nuanced and individualised assessment of an individual's emotional state. Our experiments show a considerable improvement in performance, achieving a Concordance Correlation Coefficient (CCC) of 0.475 over a baseline of 0.281.

KW - Crossmodal

KW - Mental Health

KW - Personalisation

KW - Physical Signals

KW - Transformer

UR - https://www.scopus.com/pages/publications/85186139053

U2 - 10.1109/ICDMW60847.2023.00167

DO - 10.1109/ICDMW60847.2023.00167

M3 - Conference contribution

AN - SCOPUS:85186139053

T3 - IEEE International Conference on Data Mining Workshops, ICDMW

SP - 1299

EP - 1305

BT - Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023

A2 - Wang, Jihe

A2 - He, Yi

A2 - Dinh, Thang N.

A2 - Grant, Christan

A2 - Qiu, Meikang

A2 - Pedrycz, Witold

PB - IEEE Computer Society

T2 - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023

Y2 - 1 December 2023 through 4 December 2023

ER -

Song M, Yang Z, Triantafyllopoulos A, Nakamura T, Zhang Y, Ren Z et al. Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction. In Wang J, He Y, Dinh TN, Grant C, Qiu M, Pedrycz W, editors, Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. IEEE Computer Society. 2023. p. 1299-1305. (IEEE International Conference on Data Mining Workshops, ICDMW). doi: 10.1109/ICDMW60847.2023.00167

Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this