mlsanityData to 7 Oct 2026

Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy

arXiv 2207.12141 ↗First posted 25 Jul 2022cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning (RL) often achieves higher sample efficiency in practice than model-free RL by learning a dynamics model to generate samples for policy learning. Previous works learn a dynamics model that fits under the empirical state-action visitation distribution for all historical policies, i.e., the sample replay buffer. However, in this paper, we observe that fitting the dynamics model under the distribution for all historical policies does not necessarily benefit model prediction for the current policy since the policy in use is constantly evolving over time. The evolving policy during training will cause state-action visitation distribution shifts. We theoretically analyze how this distribution shift over historical policies affects the model learning and model rollouts. We then propose a novel dynamics model learning method, named textitPolicy-adapted Dynamics Model Learning (PDML). PDML dynamically adjusts the historical policy mixture distribution to ensure the learned model can continually adapt to the state-action visitation distribution of the evolving policy. Experiments on a range of continuous control environments in MuJoCo show that PDML achieves significant improvement in sample efficiency and higher asymptotic performance combined with the state-of-the-art model-based RL methods.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 48 of its 80 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

48 of 80 references.

References · 48

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
36 more
  1. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  2. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  3. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  4. 2020Prioritized Level Replaycited by 190
  5. 2020Revisiting Fundamentals of Experience Replaycited by 171
  6. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  7. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  8. 2019Exploring Model-based Planning with Policy Networkscited by 90
  9. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  10. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  11. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  12. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  13. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  14. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 52
  15. 2018Remember and Forget for Experience Replaycited by 42
  16. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  17. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  18. 2020Bidirectional Model-based Policy Optimizationcited by 38
  19. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  20. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  21. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  22. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  23. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  24. 2015One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priorscited by 26
  25. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  26. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  27. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
  28. 2019Prioritized Sequence Experience Replaycited by 20
  29. 2022Transfer RL across Observation Feature Spaces via Model-Based Regularizationcited by 18
  30. 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
  31. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  32. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  33. 2021Large Batch Experience Replaycited by 14
  34. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  35. 2021On-Policy Model Errors in Reinforcement Learningcited by 6
  36. 2021Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 2; 2026: 2.

17 papers.

Sort:

By area:

  1. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  2. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 27
  3. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  4. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  5. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  6. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  7. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
  8. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  9. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  10. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  11. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  12. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
5 more
  1. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillationcited by 5
  2. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  3. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0
  4. 2026Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generationcited by 0
  5. 2026DT2: Decision-Targeted Digital Twinscited by 0