TY - JOUR
T1 - Affective Image Content Analysis
T2 - Two Decades Review and New Perspectives
AU - Zhao, Sicheng
AU - Yao, Xingxu
AU - Yang, Jufeng
AU - Jia, Guoli
AU - Ding, Guiguang
AU - Chua, Tat Seng
AU - Schuller, Bjorn W.
AU - Keutzer, Kurt
N1 - Publisher Copyright:
© 1979-2012 IEEE.
PY - 2022/10/1
Y1 - 2022/10/1
N2 - Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges - the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
AB - Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges - the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
KW - Affective computing
KW - emotion feature extraction
KW - emotional intelligence
KW - image emotion
KW - machine learning
UR - http://www.scopus.com/inward/record.url?scp=85112229521&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2021.3094362
DO - 10.1109/TPAMI.2021.3094362
M3 - Review article
C2 - 34214034
AN - SCOPUS:85112229521
SN - 0162-8828
VL - 44
SP - 6729
EP - 6751
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
IS - 10
ER -