# Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning

arXiv: https://arxiv.org/abs/2105.04962 · HTML: https://mlsanity.com/paper/2105.04962/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 95 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 30 of 39)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [A micro Lie theory for state estimation in robotics](https://mlsanity.com/paper/1812.01537/) (2018)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [Neural Dynamic Policies for End-to-End Sensorimotor Learning](https://mlsanity.com/paper/2012.02788/) (2020)
- [A Distributional View on Multi-Objective Policy Optimization](https://mlsanity.com/paper/2005.07513/) (2020)
- [MushroomRL: Simplifying Reinforcement Learning Research](https://mlsanity.com/paper/2001.01102/) (2020)
- [Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference](https://mlsanity.com/paper/1707.07383/) (2017)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation](https://mlsanity.com/paper/2104.13542/) (2021)
- [Riemannian Motion Policies](https://mlsanity.com/paper/1801.02854/) (2018)
- [ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flows](https://mlsanity.com/paper/2010.13129/) (2020)
- [Real-time Perception meets Reactive Motion Generation](https://mlsanity.com/paper/1703.03512/) (2017)
- [Stein Variational Model Predictive Control](https://mlsanity.com/paper/2011.07641/) (2020)
- [Residual Policy Learning for Shared Autonomy](https://mlsanity.com/paper/2004.05097/) (2020)
- [Compositional Visual Generation and Inference with Energy Based Models](https://mlsanity.com/paper/2004.06030/) (2020)
- [RMPflow: A Computational Graph for Automatic Motion Policy Generation](https://mlsanity.com/paper/1811.07049/) (2018)

## Built on by (most cited first, 16 of 16)

- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [Compositional Diffusion-Based Continuous Constraint Solvers](https://mlsanity.com/paper/2309.00966/) (2023)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Probabilistic Adaptation of Text-to-Video Models](https://mlsanity.com/paper/2306.01872/) (2023)
- [Learning Implicit Priors for Motion Optimization](https://mlsanity.com/paper/2204.05369/) (2022)
- [Compositional Generative Inverse Design](https://mlsanity.com/paper/2401.13171/) (2024)
- [Regularized Deep Signed Distance Fields for Reactive Motion Generation](https://mlsanity.com/paper/2203.04739/) (2022)
- [Multi-Modal Manipulation via Multi-Modal Policy Consensus](https://mlsanity.com/paper/2509.23468/) (2025)
- [Learning Stable Vector Fields on Lie Groups](https://mlsanity.com/paper/2110.11774/) (2021)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [Reactive Motion Generation on Learned Riemannian Manifolds](https://mlsanity.com/paper/2203.07761/) (2022)
- [Inferring Relational Potentials in Interacting Systems](https://mlsanity.com/paper/2310.14466/) (2023)
- [Latent Energy Action Planning with World Models](https://mlsanity.com/paper/2609.03294/) (2026)
- [Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning](https://mlsanity.com/paper/2505.04193/) (2025)

## Abstract (arXiv)

Reactive motion generation problems are usually solved by computing actions as a sum of policies. However, these policies are independent of each other and thus, they can have conflicting behaviors when summing their contributions together. We introduce Composable Energy Policies (CEP), a novel framework for modular reactive motion generation. CEP computes the control action by optimization over the product of a set of stochastic policies. This product of policies will provide a high probability to those actions that satisfy all the components and low probability to the others. Optimizing over the product of the policies avoids the detrimental effect of conflicting behaviors between policies choosing an action that satisfies all the objectives. Besides, we show that CEP naturally adapts to the Reinforcement Learning problem allowing us to integrate, in a hierarchical fashion, any distribution as prior, from multimodal distributions to non-smooth distributions and learn a new policy given them.