# A view on learning robust goal-conditioned value functions: Interplay between RL and MPC

arXiv: https://arxiv.org/abs/2502.06996 · HTML: https://mlsanity.com/paper/2502.06996/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 72 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 26 of 26)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement](https://mlsanity.com/paper/2002.11089/) (2020)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms](https://mlsanity.com/paper/2203.07747/) (2022)
- [Blending MPC & Value Function Approximation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2012.05909/) (2020)
- [Stability-certified reinforcement learning: A control-theoretic perspective](https://mlsanity.com/paper/1810.11505/) (2018)
- [Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models](https://mlsanity.com/paper/2006.04363/) (2020)
- [Deep Value Model Predictive Control](https://mlsanity.com/paper/1910.03358/) (2019)
- [Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents](https://mlsanity.com/paper/2410.11671/) (2024)
- [Learning from the Hindsight Plan -- Episodic MPC Improvement](https://mlsanity.com/paper/1609.09001/) (2016)
- [Learning for CasADi: Data-driven Models in Numerical Optimization](https://mlsanity.com/paper/2312.05873/) (2023)
- [Learning for MPC with Stability & Safety Guarantees](https://mlsanity.com/paper/2012.07369/) (2020)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)
- [Neural Lyapunov Differentiable Predictive Control](https://mlsanity.com/paper/2205.10728/) (2022)
- [Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior](https://mlsanity.com/paper/2310.14098/) (2023)
- [Solving robust MDPs as a sequence of static RL problems](https://mlsanity.com/paper/2410.06212/) (2024)

## Built on by (most cited first, 7 of 7)

- [MPCritic: A plug-and-play MPC architecture for reinforcement learning](https://mlsanity.com/paper/2504.01086/) (2025)
- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning](https://mlsanity.com/paper/2503.13289/) (2025)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)
- [Why Goal-Conditioned Reinforcement Learning Works: Relation to Dual Control](https://mlsanity.com/paper/2512.06471/) (2025)
- [Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies](https://mlsanity.com/paper/2604.00826/) (2026)
- [Soft MPCritic: Value-Based Reinforcement Learning for Non-Differentiable Predictive Control](https://mlsanity.com/paper/2604.01477/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) and model predictive control (MPC) offer a wealth of distinct approaches for automatic decision-making under uncertainty. Given the impact both fields have had independently across numerous domains, there is growing interest in combining the general-purpose learning capability of RL with the safety and robustness features of MPC. To this end, this paper presents a tutorial-style treatment of RL and MPC, treating them as alternative approaches to solving Markov decision processes. In our formulation, RL aims to learn a global value function through offline exploration in an uncertain environment, whereas MPC constructs a local value function through online optimization. This local-global perspective suggests new ways to design policies that combine robustness and goal-conditioned learning. Robustness is incorporated into the RL and MPC pipelines through a scenario-based approach. Goal-conditioned learning aims to alleviate the burden of engineering a reward function for RL. Combining the two leads to a single policy that unites a robust, high-level RL terminal value function with short-term, scenario-based MPC planning for reliable constraint satisfaction. This approach leverages the benefits of both RL and MPC, the effectiveness of which is demonstrated on classical control benchmarks.