The robot as scientist: Using mental simulation to test causal hypotheses extracted from human activities in virtual reality

To act effectively in its environment, a cognitive robot needs to understand the causal dependencies of all intermediate actions leading up to its goal. For example, the system has to infer that it is instrumental to open a cupboard door before trying to grasp an object inside the cupboard. In this paper, we introduce a novel learning method for extracting instrumental dependencies by following the scientific approach of observations, generation of causal hypotheses, and testing through experiments. Our method uses a virtual reality dataset containing observations from human activities to generate hypotheses about causal dependencies between actions. It detects pairs of actions with a high temporal co-occurrence and verifies if one action is instrumental in executing the other action through mental simulation in a virtual reality environment which represents the system's mental model. Our system is able to extract all present instrumental action dependencies while significantly reducing the search space for mental simulation, resulting in a 6-fold reduction in computational time.