# Learning Implicit Priors for Motion Optimization

arXiv: https://arxiv.org/abs/2204.05369 · HTML: https://mlsanity.com/paper/2204.05369/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Learning Sampling Distributions for Robot Motion Planning](https://mlsanity.com/paper/1709.05448/) (2017)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [No MCMC for me: Amortized sampling for fast and stable training of energy-based models](https://mlsanity.com/paper/2010.04230/) (2020)
- [Maximum Entropy Generators for Energy-Based Models](https://mlsanity.com/paper/1901.08508/) (2019)
- [Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference](https://mlsanity.com/paper/1707.07383/) (2017)
- [STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation](https://mlsanity.com/paper/2104.13542/) (2021)
- [ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flows](https://mlsanity.com/paper/2010.13129/) (2020)
- [An Online Learning Approach to Model Predictive Control](https://mlsanity.com/paper/1902.08967/) (2019)
- [Stein Variational Model Predictive Control](https://mlsanity.com/paper/2011.07641/) (2020)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [Multimodal Trajectory Optimization for Motion Planning](https://mlsanity.com/paper/2003.07054/) (2020)
- [Learning Sampling Distributions Using Local 3D Workspace Decompositions for Motion Planning in High Dimensions](https://mlsanity.com/paper/2010.15335/) (2020)
- [Newton methods for k-order Markov Constrained Motion Problems](https://mlsanity.com/paper/1407.0414/) (2014)
- [Functional Gradient Motion Planning in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/1601.03648/) (2016)
- [Entropy Regularized Motion Planning via Stein Variational Inference](https://mlsanity.com/paper/2107.05146/) (2021)
- [Joint Sampling and Trajectory Optimization over Graphs for Online Motion Planning](https://mlsanity.com/paper/2011.07171/) (2020)
- [Online Motion Planning Over Multiple Homotopy Classes with Gaussian Process Inference](https://mlsanity.com/paper/1908.00641/) (2019)

## Built on by (most cited first, 20 of 20)

- [Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models](https://mlsanity.com/paper/2308.01557/) (2023)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Models](https://mlsanity.com/paper/2412.19948/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [FlowMP: Learning Motion Fields for Robot Planning with Conditional Flow Matching](https://mlsanity.com/paper/2503.06135/) (2025)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [Constrained Stein Variational Trajectory Optimization](https://mlsanity.com/paper/2308.12110/) (2023)
- [Preconditioners for the Stochastic Training of Neural Fields](https://mlsanity.com/paper/2402.08784/) (2024)
- [Hierarchical Policy Blending As Optimal Transport](https://mlsanity.com/paper/2212.01938/) (2022)
- [R2-Diff: Denoising by diffusion as a refinement of retrieved motion for image-based motion prediction](https://mlsanity.com/paper/2306.09483/) (2023)
- [Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks](https://mlsanity.com/paper/2402.02551/) (2024)
- [Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulation](https://mlsanity.com/paper/2510.14643/) (2025)
- [A Unifying Variational Framework for Gaussian Process Motion Planning](https://mlsanity.com/paper/2309.00854/) (2023)
- [Global Tensor Motion Planning](https://mlsanity.com/paper/2411.19393/) (2024)
- [Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion](https://mlsanity.com/paper/2601.02873/) (2026)
- [Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities](https://mlsanity.com/paper/2603.24318/) (2026)
- [Model Tensor Planning](https://mlsanity.com/paper/2505.01059/) (2025)
- [Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation](https://mlsanity.com/paper/2604.05544/) (2026)
- [Learning to Plan from Random Exploration](https://mlsanity.com/paper/2609.38383/) (2026)

## Abstract (arXiv)

In this paper, we focus on the problem of integrating Energy-based Models (EBM) as guiding priors for motion optimization. EBMs are a set of neural networks that can represent expressive probability density distributions in terms of a Gibbs distribution parameterized by a suitable energy function. Due to their implicit nature, they can easily be integrated as optimization factors or as initial sampling distributions in the motion optimization problem, making them good candidates to integrate data-driven priors in the motion optimization problem. In this work, we present a set of required modeling and algorithmic choices to adapt EBMs into motion optimization. We investigate the benefit of including additional regularizers in the learning of the EBMs to use them with gradient-based optimizers and we present a set of EBM architectures to learn generalizable distributions for manipulation tasks. We present multiple cases in which the EBM could be integrated for motion optimization and evaluate the performance of learned EBMs as guiding priors for both simulated and real robot experiments.