# Model Based Planning with Energy Based Models

arXiv: https://arxiv.org/abs/1909.06878 · HTML: https://mlsanity.com/paper/1909.06878/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [PoCo: Policy Composition from and for Heterogeneous Robot Learning](https://mlsanity.com/paper/2402.02511/) (2024)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Adaptive Online Replanning with Diffusion Models](https://mlsanity.com/paper/2310.09629/) (2023)
- [Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement](https://mlsanity.com/paper/2304.14391/) (2023)
- [Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement](https://mlsanity.com/paper/2307.04751/) (2023)
- [Model-Based Reinforcement Learning via Latent-Space Collocation](https://mlsanity.com/paper/2106.13229/) (2021)
- [Compositional Generative Inverse Design](https://mlsanity.com/paper/2401.13171/) (2024)
- [A survey and taxonomy of loss functions in machine learning](https://mlsanity.com/paper/2301.05579/) (2023)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recovery](https://mlsanity.com/paper/2511.05007/) (2025)
- [Energy-based Potential Games for Joint Motion Forecasting and Control](https://mlsanity.com/paper/2312.01811/) (2023)
- [Planning with Sequence Models through Iterative Energy Minimization](https://mlsanity.com/paper/2303.16189/) (2023)
- [SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models](https://mlsanity.com/paper/2512.05955/) (2025)
- [Pseudo-Spherical Contrastive Divergence](https://mlsanity.com/paper/2111.00780/) (2021)
- [Follow the Energy, Find the Path: Riemannian Metrics from Energy-Based Models](https://mlsanity.com/paper/2505.18230/) (2025)
- [Inferring Relational Potentials in Interacting Systems](https://mlsanity.com/paper/2310.14466/) (2023)
- [On a Connection between Differential Games, Optimal Control, and Energy-based Models for Multi-Agent Interactions](https://mlsanity.com/paper/2308.16539/) (2023)
- [Implicit Two-Tower Policies](https://mlsanity.com/paper/2208.01191/) (2022)
- [Energy-Based Models for Cross-Modal Localization using Convolutional Transformers](https://mlsanity.com/paper/2306.04021/) (2023)
- [Learning Energy-Based Generative Models via Potential Flow: A Variational Principle Approach to Probability Density Homotopy Matching](https://mlsanity.com/paper/2504.16262/) (2025)
- [Maximum entropy exploration in contextual bandits with neural networks and energy based models](https://mlsanity.com/paper/2210.06302/) (2022)
- [Deep Generative Models for Decision-Making and Control](https://mlsanity.com/paper/2306.08810/) (2023)
- [A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models](https://mlsanity.com/paper/2512.18730/) (2025)
- [Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modelling](https://mlsanity.com/paper/2407.15238/) (2024)

## Abstract (arXiv)

Model-based planning holds great promise for improving both sample efficiency and generalization in reinforcement learning (RL). We show that energy-based models (EBMs) are a promising class of models to use for model-based planning. EBMs naturally support inference of intermediate states given start and goal state distributions. We provide an online algorithm to train EBMs while interacting with the environment, and show that EBMs allow for significantly better online learning than corresponding feed-forward networks. We further show that EBMs support maximum entropy state inference and are able to generate diverse state space plans. We show that inference purely in state space - without planning actions - allows for better generalization to previously unseen obstacles in the environment and prevents the planner from exploiting the dynamics model by applying uncharacteristic action sequences. Finally, we show that online EBM training naturally leads to intentionally planned state exploration which performs significantly better than random exploration.