# Information Theoretic Model Predictive Q-Learning

arXiv: https://arxiv.org/abs/2001.02153 · HTML: https://mlsanity.com/paper/2001.02153/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2019.

## Built on by (most cited first, 18 of 18)

- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks](https://mlsanity.com/paper/2109.10312/) (2021)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [Learning Sampling Distributions for Model Predictive Control](https://mlsanity.com/paper/2212.02587/) (2022)
- [The Value of Planning for Infinite-Horizon Model Predictive Control](https://mlsanity.com/paper/2104.02863/) (2021)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Residual-MPPI: Online Policy Customization for Continuous Control](https://mlsanity.com/paper/2407.00898/) (2024)
- [Continuous Monte Carlo Graph Search](https://mlsanity.com/paper/2210.01426/) (2022)
- [CoVO-MPC: Theoretical Analysis of Sampling-based MPC and Optimal Covariance Design](https://mlsanity.com/paper/2401.07369/) (2024)
- [Model Predictive Adversarial Imitation Learning for Planning from Observation](https://mlsanity.com/paper/2507.21533/) (2025)
- [Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making](https://mlsanity.com/paper/2512.17091/) (2025)
- [Model-Free Characterizations of the Hamilton-Jacobi-Bellman Equation and Convex Q-Learning in Continuous Time](https://mlsanity.com/paper/2210.08131/) (2022)
- [Goal-Conditioned Terminal Value Estimation for Real-time and Multi-task Model Predictive Control](https://mlsanity.com/paper/2410.04929/) (2024)
- [Soft MPCritic: Value-Based Reinforcement Learning for Non-Differentiable Predictive Control](https://mlsanity.com/paper/2604.01477/) (2026)
- [RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments](https://mlsanity.com/paper/2609.39854/) (2026)

## Abstract (arXiv)

Model-free Reinforcement Learning (RL) works well when experience can be collected cheaply and model-based RL is effective when system dynamics can be modeled accurately. However, both assumptions can be violated in real world problems such as robotics, where querying the system can be expensive and real-world dynamics can be difficult to model. In contrast to RL, Model Predictive Control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models. We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real systems in a systematic manner.