# Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization

arXiv: https://arxiv.org/abs/2004.08763 · HTML: https://mlsanity.com/paper/2004.08763/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2020.

## Built on by (most cited first, 30 of 32)

- [Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models](https://mlsanity.com/paper/2502.14819/) (2025)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Physical Design using Differentiable Learned Simulators](https://mlsanity.com/paper/2202.00728/) (2022)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models](https://mlsanity.com/paper/2605.06388/) (2026)
- [Gradient-based Planning with World Models](https://mlsanity.com/paper/2312.17227/) (2023)
- [Neuro-algorithmic Policies enable Fast Combinatorial Generalization](https://mlsanity.com/paper/2102.07456/) (2021)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Closing the Train-Test Gap in World Models for Gradient-Based Planning](https://mlsanity.com/paper/2512.09929/) (2025)
- [CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2112.07746/) (2021)
- [Gradient-Based Trajectory Optimization With Learned Dynamics](https://mlsanity.com/paper/2204.04558/) (2022)
- [Parallel Stochastic Gradient-Based Planning for World Models](https://mlsanity.com/paper/2602.00475/) (2026)
- [Physics-informed Temporal Difference Metric Learning for Robot Motion Planning](https://mlsanity.com/paper/2505.05691/) (2025)
- [A Simple Decentralized Cross-Entropy Method](https://mlsanity.com/paper/2212.08235/) (2022)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations](https://mlsanity.com/paper/2203.02857/) (2022)
- [PcLast: Discovering Plannable Continuous Latent States](https://mlsanity.com/paper/2311.03534/) (2023)
- [IMWM: Intuition Models Complement World Models for Latent Planning](https://mlsanity.com/paper/2606.01626/) (2026)
- [Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method](https://mlsanity.com/paper/2103.12020/) (2021)
- [Preference-Optimized Pareto Set Learning for Blackbox Optimization](https://mlsanity.com/paper/2408.09976/) (2024)
- [Lifting Embodied World Models for Planning and Control](https://mlsanity.com/paper/2604.26182/) (2026)
- [Conditional Diffusion with Less Explicit Guidance via Model Predictive Control](https://mlsanity.com/paper/2210.12192/) (2022)
- [Safe Beyond the Horizon: Efficient Sampling-based MPC with Neural Control Barrier Functions](https://mlsanity.com/paper/2502.15006/) (2025)
- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination](https://mlsanity.com/paper/2605.04568/) (2026)
- [NeuroSMPC: A Neural Network guided Sampling Based MPC for On-Road Autonomous Driving](https://mlsanity.com/paper/2310.13077/) (2023)
- [Graph Neural Networks with Model-based Reinforcement Learning for Multi-agent Systems](https://mlsanity.com/paper/2407.09249/) (2024)
- [Cross-Entropy Games and Frost Training](https://mlsanity.com/paper/2605.27701/) (2026)

## Abstract (arXiv)

Recent works in high-dimensional model-predictive control and model-based reinforcement learning with learned dynamics and reward models have resorted to population-based optimization methods, such as the Cross-Entropy Method (CEM), for planning a sequence of actions. To decide on an action to take, CEM conducts a search for the action sequence with the highest return according to the dynamics model and reward. Action sequences are typically randomly sampled from an unconditional Gaussian distribution and evaluated on the environment. This distribution is iteratively updated towards action sequences with higher returns. However, this planning method can be very inefficient, especially for high-dimensional action spaces. An alternative line of approaches optimize action sequences directly via gradient descent, but are prone to local optima. We propose a method to solve this planning problem by interleaving CEM and gradient descent steps in optimizing the action sequence. Our experiments show faster convergence of the proposed hybrid approach, even for high-dimensional action spaces, avoidance of local minima, and better or equal performance to CEM. Code accompanying the paper is available here https://github.com/homangab/gradcem.