Motion prediction of virtual patterns, human hand motions, and a simplified hand manipulation task with hierarchical temporal memory

In this paper we utilize Numenta's Hierarchical Temporal Memory implementation NuPIC for online visual motion pattern prediction and test its performance on virtual animations as well as real world human motion data. For evaluation we run a series of progressively more complex experiments testing specific capabilities: Prediction of fixed-time noise-free motion animations, prediction of protocol-directed tasks with real-world camera captured human motion data, and lastly prediction of repetitive tasks performed without a strict protocol. Results show that the presented setup is able to predict time sequenced images as well as highly variable human motions increasingly well over several iterations. Limits are faced for non sequential variable hand motion execution: Here, predictions are made but do not improve in quality over time. The network runs online in real time and can be transferred to different tasks without expert knowledge. These characteristics qualify the setup for human robot interaction scenarios without the need for verified prediction accuracy.