Emotion recognition in the manual interaction with graphical user interfaces

In this paper we introduce a novel approach to human emotion recognition based on manual computer interaction. The presented methods rely on conventional graphical input devices: Firstly a standard mouse as used on desktop PCs, and secondly the interaction on touch-screens or -pads as in public information terminals, palm-top devices or tablet PCs is considered. Additionally the gain of the integration of touch pressure information is evaluated. Four discrete emotional states are classified: irritation, annoyance, reflectiveness, and neutral affect for the use in initiative tutoring, error clarification, Internet customer personalization, and others. The optimal feature-set is discussed and ranked according to a linear discriminant analysis. A working system using Support Vector Machines for the classification is tested in real-life scenarios. The performance of up to 83.2% correct assignment clearly indicates that user emotion recognition is possible without special hardware in any standard graphical user environment independent of the underlying application.