mlsanityData to 7 Oct 2026

Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planning

arXiv 2406.19561 ↗First posted 27 Jun 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We study how a Reinforcement Learning (RL) system can remain sample-efficient when learning from an imperfect model of the environment. This is particularly challenging when the learning system is resource-constrained and in continual settings, where the environment dynamics change. To address these challenges, our paper introduces an online, meta-gradient algorithm that tunes a probability with which states are queried during Dyna-style planning. Our study compares the aggregate, empirical performance of this meta-gradient method to baselines that employ conventional sampling strategies. Results indicate that our method improves efficiency of the planning process, which, as a consequence, improves the sample-efficiency of the overall learning process. On the whole, we observe that our meta-learned solutions avoid several pathologies of conventional planning approaches, such as sampling inaccurate transitions and those that stall credit assignment. We believe these findings could prove useful, in future work, for designing model-based RL systems at scale.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 44 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

26 of 44 references.

References · 26

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  8. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  9. 2018Meta-Gradient Reinforcement Learningcited by 156
  10. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  11. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  12. 2016The Predictron: End-To-End Learning and Planningcited by 92
14 more
  1. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  2. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  3. 2021Bootstrapped Meta-Learningcited by 45
  4. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 42
  5. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
  6. 2021Proper Value Equivalencecited by 33
  7. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  8. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  9. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  10. 2022The Alberta Plan for AI Researchcited by 17
  11. 2020Frequency-based Search-control in Dynacited by 7
  12. 2023On the Convergence of Bounded Agentscited by 7
  13. 2021Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planningcited by 3
  14. 2022Should Models Be Accurate?cited by 2

Built on by

Not cited yet.