mlsanityData to 7 Oct 2026

Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation

arXiv 2106.03273 ↗First posted 6 Jun 2021cs.LG cs.AI stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 29 of its 85 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

29 of 85 references.

References · 29

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2021Convex optimizationcited by 2,479
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  12. 2019Deep Equilibrium Modelscited by 603
17 more
  1. 2019Meta-Learning with Implicit Gradientscited by 523
  2. 2019Model-Based Reinforcement Learning for Ataricited by 495
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  4. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  5. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  6. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  10. 2016Value Iteration Networkscited by 158
  11. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  12. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  13. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  14. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  15. 2019The Differentiable Cross-Entropy Methodcited by 26
  16. 2020Policy-Aware Model Learning for Policy Gradient Methodscited by 23
  17. 2019Gradient-Aware Model-based Policy Searchcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 6; 2024: 8; 2025: 7; 2026: 4.

35 papers.

Sort:

By area:

  1. 2023A Survey of Contextual Optimization Methods for Decision Making under Uncertaintycited by 81
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  3. 2024Mastering Memory Tasks with World Modelscited by 38
  4. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  5. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  6. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  7. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  8. 2024Scaling physics-informed hard constraints with mixture-of-expertscited by 24
  9. 2024Functional Bilevel Optimization for Machine Learningcited by 20
  10. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  11. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  12. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 13
23 more
  1. 2023TaskMet: Task-Driven Metric Learning for Model Learningcited by 11
  2. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  3. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  4. 2022Operator Splitting Value Iterationcited by 8
  5. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  6. 2022Scaling up and Stabilizing Differentiable Planning with Implicit Differentiationcited by 6
  7. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  8. 2024DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learningcited by 5
  9. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  10. 2025Learning Theory for Kernel Bilevel Optimizationcited by 2
  11. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  12. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  13. 2026Non-Stationary Functional Bilevel Optimizationcited by 2
  14. 2023Decision-Focused Model-based Reinforcement Learning for Reward Transfercited by 1
  15. 2023λ-models: Effective Decision-Aware Reinforcement Learning with Latent Modelscited by 1
  16. 2024Learning Set Functions with Implicit Differentiationcited by 1
  17. 2025Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimalitycited by 1
  18. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1
  19. 2024Efficient Public Health Intervention Planning Using Decomposition-Based Decision-Focused Learningcited by 0
  20. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0
  21. 2026Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communicationcited by 0
  22. 2026IGT-OMD: Implicit Gradient Transport for Decision-Focused Learning under Delayed Feedbackcited by 0
  23. 2026DT2: Decision-Targeted Digital Twinscited by 0