Automatic Recognition of Human Activities Combining Model-based AI and Machine Learning

Constantin Patsch, Marsil Zakour, Rahul Chaudhari

Research output: Contribution to journalConference articlepeer-review

1 Scopus citations


Developing intelligent assistants for activities of daily living (ADL) is an important topic in eldercare due to the aging society in industrialized countries. Recognizing activities and understanding the human’s intended goal are the major challenges associated with such a system. We propose a hybrid model for composite activity recognition in a household environment by combining Machine Learning and knowledge-based models. The Machine Learning part, based on structural Recurrent Neural Networks (S-RNN), performs low-level activity recognition based on video data. The knowledge-based part, based on our extended Activation Spreading Network architecture, models and recognizes the contextual meaning of an activity within a plan structure. This model is able to recognize activities, underlying goals and sub-goals, and is able to predict subsequent activities. Evaluating our action S-RNN on data from the 3D activity simulator HOIsim yields a macro average F1 score of 0.97 and an accuracy of 0.99. The hybrid model is evaluated with activation value graphs.

Original languageEnglish
Pages (from-to)15-22
Number of pages8
JournalInternational Conference on Agents and Artificial Intelligence
StatePublished - 2022
Event14th International Conference on Agents and Artificial Intelligence , ICAART 2022 - Virtual, Online
Duration: 3 Feb 20225 Feb 2022


  • Activity and Plan Recognition
  • Knowledge Representation and Reasoning
  • Machine Learning


Dive into the research topics of 'Automatic Recognition of Human Activities Combining Model-based AI and Machine Learning'. Together they form a unique fingerprint.

Cite this