Safe Policy Improvement Approaches and Their Limitations

Philipp Scholl, Felix Dietrich, Clemens Otte, Steffen Udluft

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Safe Policy Improvement (SPI) is an important technique for offline reinforcement learning in safety critical applications as it improves the behavior policy with a high probability. We classify various SPI approaches from the literature into two groups, based on how they utilize the uncertainty of state-action pairs. Focusing on the Soft-SPIBB (Safe Policy Improvement with Soft Baseline Bootstrapping) algorithms, we show that their claim of being provably safe does not hold. Based on this finding, we develop adaptations, the Adv-Soft-SPIBB algorithms, and show that they are provably safe. A heuristic adaptation, Lower-Approx-Soft-SPIBB, yields the best performance among all SPIBB algorithms in extensive experiments on two benchmarks. We also check the safety guarantees of the provably safe algorithms and show that huge amounts of data are necessary such that the safety bounds become useful in practice.

Original languageEnglish
Title of host publicationAgents and Artificial Intelligence - 14th International Conference, ICAART 2022, Revised Selected Papers
EditorsAna Paula Rocha, Luc Steels, Jaap van den Herik
PublisherSpringer Science and Business Media Deutschland GmbH
Pages74-98
Number of pages25
ISBN (Print)9783031229527
DOIs
StatePublished - 2022
Event14th International Conference on Agents and Artificial Intelligence, ICAART 2022 - Virtual, Online
Duration: 3 Feb 20225 Feb 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13786 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th International Conference on Agents and Artificial Intelligence, ICAART 2022
CityVirtual, Online
Period3/02/225/02/22

Keywords

  • Markov decision processes
  • Risk-sensitive reinforcement learning
  • Safe policy improvement

Fingerprint

Dive into the research topics of 'Safe Policy Improvement Approaches and Their Limitations'. Together they form a unique fingerprint.

Cite this