mlsanityData to 7 Oct 2026

A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning

arXiv 2310.06253 ↗First posted 10 Oct 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Model-based Reinforcement Learning (MBRL) aims to make agents more sample-efficient, adaptive, and explainable by learning an explicit model of the environment. While the capabilities of MBRL agents have significantly improved in recent years, how to best learn the model is still an unresolved question. The majority of MBRL algorithms aim at training the model to make accurate predictions about the environment and subsequently using the model to determine the most rewarding actions. However, recent research has shown that model predictive accuracy is often not correlated with action quality, tracing the root cause to the objective mismatch between accurate dynamics model learning and policy optimization of rewards. A number of interrelated solution categories to the objective mismatch problem have emerged as MBRL continues to mature as a research area. In this work, we provide an in-depth survey of these solution categories and propose a taxonomy to foster future research.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 101 of its 141 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

101 of 141 references.

References · 101

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  12. 2023Mastering Diverse Domains through World Modelscited by 1,193
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2018DeepMind Control Suitecited by 935
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  13. 2017Smart "Predict, then Optimize"cited by 332
  14. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  15. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  16. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  17. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  18. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  19. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  20. 2018Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimizationcited by 220
  21. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  22. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  23. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  24. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  25. 2016Value Iteration Networkscited by 158
  26. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  27. 2019Behaviour Suite for Reinforcement Learningcited by 129
  28. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  29. 2017Value Prediction Networkcited by 117
  30. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  31. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  32. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  33. 2016The Predictron: End-To-End Learning and Planningcited by 92
  34. 2020Active inference on discrete state-spaces: a synthesiscited by 91
  35. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  36. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  37. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  38. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 4; 2026: 3.

9 papers.

  1. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  2. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  3. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 9
  4. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  5. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistencycited by 2
  6. 2025Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimalitycited by 1
  7. 2025Safe Planning and Policy Optimization via World Model Learningcited by 0
  8. 2025CORL: Reinforcement Learning of MILP Policies Solved via Branch and Boundcited by 0
  9. 2026Same World, Different Knowledge: When Isolated Audits Misjudge World-Model Repairscited by 0