Bypassing the Simulation-to-Reality Gap: Online Reinforcement Learning Using a Supervisor

Benjamin David Evans, Johannes Betz, Hongrui Zheng, Herman A. Engelbrecht, Rahul Mangharam, Hendrik W. Jordaan

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Deep reinforcement learning (DRL) is a promising method to learn control policies for robots only from demonstration and experience. To cover the whole dynamic behaviour of the robot, DRL training is an active exploration process typically performed in simulation environments. Although this simulation training is cheap and fast, applying DRL algorithms to real-world settings is difficult. If agents are trained until they perform safely in simulation, transferring them to physical systems is difficult due to the sim-to-real gap caused by the difference between the simulation dynamics and the physical robot. In this paper, we present a method of online training a DRL agent to drive autonomously on a physical vehicle by using a model-based safety supervisor. Our solution uses a supervisory system to check if the action selected by the agent is safe or unsafe and ensure that a safe action is always implemented on the vehicle. With this, we can bypass the sim-to-real problem while training the DRL algorithm safely, quickly, and efficiently. We compare our method with conventional learning in simulation and on a physical vehicle. We provide a variety of real-world experiments where we train online a small-scale vehicle to drive autonomously with no prior simulation training. The evaluation results show that our method trains agents with improved sample efficiency while never crashing, and the trained agents demonstrate better driving performance than those trained in simulation.

Original languageEnglish
Title of host publication2023 21st International Conference on Advanced Robotics, ICAR 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages325-331
Number of pages7
ISBN (Electronic)9798350342291
DOIs
StatePublished - 2023
Event21st International Conference on Advanced Robotics, ICAR 2023 - Abu Dhabi, United Arab Emirates
Duration: 5 Dec 20238 Dec 2023

Publication series

Name2023 21st International Conference on Advanced Robotics, ICAR 2023

Conference

Conference21st International Conference on Advanced Robotics, ICAR 2023
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period5/12/238/12/23

Fingerprint

Dive into the research topics of 'Bypassing the Simulation-to-Reality Gap: Online Reinforcement Learning Using a Supervisor'. Together they form a unique fingerprint.

Cite this