Hierarchical Reinforcement Learning for Waypoint-based Exploration in Robotic Devices

The training of Deep Reinforcement Learning algorithms on robotic devices is challenging due to their large number of actuators and limited number of feasible action sequences. This paper addresses this challenge by extending and transferring existing approaches for waypoint-based exploration with Hierarchical Reinforcement Learning to the domain of robotic devices. The resulting algorithm utilizes a top-level policy, which suggests waypoints to a bottom-level policy that controls the system actuators. The waypoints can either be provided to the top-level policy as domain knowledge or be learned from scratch. The algorithm explicitly accounts for the low number of feasible waypoints and waypoint transitions that are characteristic of robotic devices. The effectiveness of the approach is evaluated on the simulation of a research demonstrator, and a separate ablation study proves the importance of its components.