Context-Based Meta-Reinforcement Learning with Bayesian Nonparametric Models

Zhenshan Bing, Yuqi Yun, Kai Huang, Alois Knoll

Research output: Contribution to journalArticlepeer-review

Abstract

Deep reinforcement learning agents usually need to collect a large number of interactions to solve a single task. In contrast, meta-reinforcement learning (meta-RL) aims to quickly adapt to new tasks using a small amount of experience by leveraging the knowledge from training on a set of similar tasks. State-of-the-art context-based meta-RL algorithms use the context to encode the task information and train a policy conditioned on the inferred latent task encoding. However, most recent works are limited to parametric tasks, where a handful of variables control the full variation in the task distribution, and also failed to work in non-stationary environments due to the few-shot adaptation setting. To address those limitations, we propose <bold>ME</bold>ta-reinforcement <bold>L</bold>earning with <bold>T</bold>ask <bold>S</bold>elf-discovery (MELTS), which adaptively learns qualitatively different nonparametric tasks and adapts to new tasks in a zero-shot manner. We introduce a novel deep clustering framework (DPMM-VAE) based on an infinite mixture of Gaussians, which combines the Dirichlet process mixture model (DPMM) and the variational autoencoder (VAE), to simultaneously learn task representations and cluster the tasks in a self-adaptive way. Integrating DPMM-VAE into MELTS enables it to adaptively discover the multi-modal structure of the nonparametric task distribution, which previous methods using isotropic Gaussian random variables cannot model. In addition, we propose a zero-shot adaptation mechanism and a recurrence-based context encoding strategy to improve the data efficiency and make our algorithm applicable in non-stationary environments. On various continuous control tasks with both parametric and nonparametric variations, our algorithm produces a more structured and self-adaptive task latent space and also achieves superior sample efficiency and asymptotic performance compared with state-of-the-art meta-RL algorithms.

Original languageEnglish
Pages (from-to)1-18
Number of pages18
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
DOIs
StateAccepted/In press - 2024

Keywords

  • Adaptation models
  • Bayesian nonparametric model
  • Clustering algorithms
  • Inference algorithms
  • Markov decision processes
  • Meta-reinforcement learning
  • Optimization
  • Task analysis
  • Training
  • robotic control
  • task adaptation
  • task inference

Fingerprint

Dive into the research topics of 'Context-Based Meta-Reinforcement Learning with Bayesian Nonparametric Models'. Together they form a unique fingerprint.

Cite this