mlsanityData to 7 Oct 2026

Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents

arXiv 2405.16899 ↗First posted 27 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In neuroscience, one of the key behavioral tests for determining whether a subject of study exhibits model-based behavior is to study its adaptiveness to local changes in the environment. In reinforcement learning, however, recent studies have shown that modern model-based agents display poor adaptivity to such changes. The main reason for this is that modern agents are typically designed to improve sample efficiency in single task settings and thus do not take into account the challenges that can arise in other settings. In local adaptation settings, one particularly important challenge is in quickly building and maintaining a sufficiently accurate model after a local change. This is challenging for deep model-based agents as their models and replay buffers are monolithic structures lacking distribution shift handling capabilities. In this study, we show that the conceptually simple idea of partial models can allow deep model-based agents to overcome this challenge and thus allow for building locally adaptive model-based agents. By modeling the different parts of the state space through different models, the agent can not only maintain a model that is accurate across the state space, but it can also quickly adapt it in the presence of a local change in the environment. We demonstrate this by showing that the use of partial models in agents such as deep Dyna-Q, PlaNet and Dreamer can allow for them to effectively adapt to the local changes in their environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

20 of 34 references.

References · 20

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2018DeepMind Control Suitecited by 935
  7. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  8. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  9. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  10. 2019When to use parametric models in reinforcement learning?cited by 118
  11. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  12. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 41
8 more
  1. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  2. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  3. 2020The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learningcited by 11
  4. 2021Temporally Abstract Partial Modelscited by 11
  5. 2022Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methodscited by 9
  6. 2023Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learningcited by 6
  7. 2022A Look at Value-Based Decision-Time vs. Background Planning Methods Across Different Settingscited by 5
  8. 2023Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learningcited by 2

Built on by

Not cited yet.