mlsanityData to 14 Oct 2026

Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents

arXiv 2507.13491 ↗First posted 17 Jul 2025cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Training sophisticated agents for optimal decision-making under uncertainty has been key to the rapid development of modern autonomous systems across fields. Notably, model-free reinforcement learning (RL) has enabled decision-making agents to improve their performance directly through system interactions, with minimal prior knowledge about the system. Yet, model-free RL has generally relied on agents equipped with deep neural network function approximators, appealing to the networks' expressivity to capture the agent's policy and value function for complex systems. However, neural networks amplify the issues of sample inefficiency, unsafe learning, and limited interpretability in model-free RL. To this end, this work introduces model-based agents as a compelling alternative for control policy approximation, leveraging adaptable models of system dynamics, cost, and constraints for safe policy learning. These models can encode prior system knowledge to inform, constrain, and aid in explaining the agent's decisions, while deficiencies due to model mismatch can be remedied with model-free RL. We outline the benefits and challenges of learning model-based agents -- exemplified by model predictive control -- and detail the primary learning approaches: Bayesian optimization, policy search RL, and offline strategies, along with their respective strengths. While model-free RL has long been established, its interplay with model-based agents remains largely unexplored, motivating our perspective on their combined potentials for sample-efficient learning of safe and interpretable decision-making agents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 108 of its 195 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

108 of 195 references.

References · 108

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
38 more
  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  12. 2015Deep Kernel Learningcited by 445
  13. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 374
  14. 2019Control Barrier Functions: Theory and Applicationscited by 372
  15. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  16. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  17. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  18. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  19. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  20. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  21. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  22. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  23. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  24. 2018Differentiable MPC for End-to-end Planning and Controlcited by 189
  25. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 182
  26. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 179
  27. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  28. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  29. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  30. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  31. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 149
  32. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  33. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 141
  34. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  35. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  36. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  37. 2016Off-policy evaluation for slate recommendationcited by 103
  38. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101

50 most cited of 108 All 108 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards General-Purpose Model-Free Reinforcement Learning
  2. 2024Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions
  3. 2024Reset-free Reinforcement Learning with World Models
  4. 2019Reinforcement learning with world model
  5. 2024Guiding Reinforcement Learning with Incomplete System Dynamics
  6. 2024A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments
  7. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  8. 2025Control-Optimized Deep Reinforcement Learning for Artificially Intelligent Autonomous Systems