Learning Causal Relationships of Object Properties and Affordances Through Human Demonstrations and Self-Supervised Intervention for Purposeful Action in Transfer Environments

Learning object affordances enables robots to plan and perform purposeful actions. However, a fundamental challenge for the utilization of affordance knowledge lies in its generalization to unknown objects and environments. In this letter we present a new method for learning causal relationships between object properties and object affordances which can be transferred to other environments. Our approach, implemented on a PR2 robot, generates hypotheses of property-affordance models in a toy environment based on human demonstrations that are subsequently tested through interventional experiments. The system relies on information theory to choose experiments for maximal information gain, performs them self-supervised and uses the observed outcome to iteratively refine the set of candidate causal models. The learned causal knowledge is human-interpretable in the form of graphical models, stored in the knowledge graph. We validate our method through a task requiring affordance knowledge transfer to three different unknown environments. Our results show that extending learning from human demonstrations by causal learning through interventions led to a 71.7% decrease in model uncertainty and improved affordance classification in the transfer environments on average by 47.49%.