mlsanityData to 14 Oct 2026

Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning

arXiv 2503.13289 ↗First posted 17 Mar 2025eess.SY cs.SYcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Making optimal decisions under uncertainty is a shared problem among distinct fields. While optimal control is commonly studied in the framework of dynamic programming, it is approached with differing perspectives of the Bellman optimality condition. In one perspective, the Bellman equation is used to derive a global optimality condition useful for iterative learning of control policies through interactions with an environment. Alternatively, the Bellman equation is also widely adopted to derive tractable optimization-based control policies that satisfy a local notion of optimality. By leveraging ideas from the two perspectives, we present a local-global paradigm for optimal control suited for learning interpretable local decision makers that approximately satisfy the global Bellman equation. The benefits and practical complications in local-global learning are discussed. These aspects are exemplified through case studies, which give an overview of two distinct strategies for unifying reinforcement learning and model predictive control. We discuss the challenges and trade-offs in these local-global strategies, towards highlighting future research opportunities for safe and optimal decision-making under uncertainty.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 64 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

24 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption29 May

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  10. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  12. 2018Differentiable MPC for End-to-end Planning and Controlcited by 189
11 more
  1. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  2. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  3. 2019Safe Reinforcement Learning Using Robust MPCcited by 85
  4. 2019Data-driven Economic NMPC using Reinforcement Learningcited by 68
  5. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 32
  6. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 26
  7. 2019Deep Value Model Predictive Controlcited by 20
  8. 2023Learning for CasADi: Data-driven Models in Numerical Optimizationcited by 15
  9. 2020Learning for MPC with Stability & Safety Guaranteescited by 14
  10. 2024PC-Gym: Benchmark Environments For Process Control Problemscited by 10
  11. 2025A view on learning robust goal-conditioned value functions: Interplay between RL and MPCcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scale
  2. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  3. 2025Theoretical Barriers in Bellman-Based Reinforcement Learning
  4. 2025A Temporal Difference Method for Stochastic Continuous Dynamics
  5. 2026Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence
  6. 2026Generalised Bellman recurrence and three dualities in sequential decision-making
  7. 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
  8. 2024Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions