mlsanityData to 7 Oct 2026

A Game Theoretic Framework for Model Based Reinforcement Learning

arXiv 2004.07804 ↗First posted 16 Apr 2020cs.LG cs.AI cs.ROcited by 72

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 5 of its 88 references that are in the map. First posted on arXiv 16 Apr 2020.

Builds on

5 of 88 references.

Closest ancestors · 4

Same direction, newest first

  1. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  2. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  4. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb

Other references · 1

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 25; 2022: 19; 2023: 10; 2024: 8; 2025: 5; 2026: 3.

72 papers.

Sort:

By area:

  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  3. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  6. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  7. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  8. 2022A Survey on Model-based Reinforcement Learningcited by 65
  9. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  10. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  11. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  12. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
38 more
  1. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  2. 2021Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiationcited by 35
  3. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  4. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  5. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
  6. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  7. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  8. 2022Performative Reinforcement Learningcited by 25
  9. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  10. 2021Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?cited by 24
  11. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  12. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  13. 2021Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoirescited by 20
  14. 2021Learning Modular Robot Control Policiescited by 18
  15. 2021Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithmscited by 18
  16. 2022A Ranking Game for Imitation Learningcited by 18
  17. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  18. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  19. 2021Minimax Model Learningcited by 16
  20. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  21. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  22. 2021Discretization Drift in Two-Player Gamescited by 14
  23. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  24. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  25. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  26. 2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
  27. 2023Causal Semantic Communication for Digital Twins: A Generalizable Imitation Learning Approachcited by 14
  28. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  29. 2024Performative Reinforcement Learning in Gradually Shifting Environmentscited by 13
  30. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  31. 2021Complex Momentum for Optimization in Gamescited by 11
  32. 2021Safe Chance Constrained Reinforcement Learning for Batch Process Controlcited by 10
  33. 2022Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimizationcited by 10
  34. 2021Characterizing the Gap Between Actor-Critic and Policy Gradientcited by 9
  35. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  36. 2024BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Modelscited by 7
  37. 2023Learning non-Markovian Decision-Making from State-only Sequencescited by 6
  38. 2022Offline Reinforcement Learning for Road Traffic Controlcited by 5

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stackelberg Batch Policy Learning
  2. 2020Ready Policy One: World Building Through Active Learning
  3. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  4. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  5. 2026Reinforcement Learning: From Algorithms To Foundation Models
  6. 2023A Survey on Offline Model-Based Reinforcement Learning
  7. 2026Maximum Likelihood Reinforcement Learning
  8. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games