# Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning

arXiv: https://arxiv.org/abs/2505.04193 · HTML: https://mlsanity.com/paper/2505.04193/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 7 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 31)

- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity](https://mlsanity.com/paper/2208.05767/) (2022)
- [InfoBot: Transfer and Exploration via the Information Bottleneck](https://mlsanity.com/paper/1901.10902/) (2019)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation](https://mlsanity.com/paper/2310.16050/) (2023)
- [Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation](https://mlsanity.com/paper/2307.08875/) (2023)
- [Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2006.13704/) (2020)
- [Robust Predictable Control](https://mlsanity.com/paper/2109.03214/) (2021)
- [Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration](https://mlsanity.com/paper/2305.19476/) (2023)
- [Leveraging Symmetries in Pick and Place](https://mlsanity.com/paper/2308.07948/) (2023)
- [Robust Quadrupedal Locomotion via Risk-Averse Policy Learning](https://mlsanity.com/paper/2308.09405/) (2023)
- [Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning](https://mlsanity.com/paper/1909.05950/) (2019)
- [R'enyi State Entropy for Exploration Acceleration in Reinforcement Learning](https://mlsanity.com/paper/2203.04297/) (2022)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images](https://mlsanity.com/paper/2203.01810/) (2022)
- [Reinforcement Learning with Simple Sequence Priors](https://mlsanity.com/paper/2305.17109/) (2023)
- [Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditions](https://mlsanity.com/paper/2312.16438/) (2023)
- [Entropy Maximization for Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2105.07490/) (2021)
- [PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks](https://mlsanity.com/paper/2307.05891/) (2023)
- [Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors](https://mlsanity.com/paper/2410.17551/) (2024)
- [Predictive Modeling of Periodic Behavior for Human-Robot Symbiotic Walking](https://mlsanity.com/paper/2005.13139/) (2020)

## Abstract (arXiv)

Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive results of deep reinforcement learning in complex control tasks, it is prone to capturing intricate and spurious correlations between observations and actions, leading to failure under slight perturbations to the environment. To tackle this problem, in this work we introduce a novel inductive bias towards simple policies in reinforcement learning. The simplicity inductive bias is introduced by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories. Our reinforcement learning agent, Trajectory Entropy Reinforcement Learning, is optimized to minimize the trajectory entropy while maximizing rewards. We show that the trajectory entropy can be effectively estimated by learning a variational parameterized action prediction model, and use the prediction model to construct an information-regularized reward function. Furthermore, we construct a practical algorithm that enables the joint optimization of models, including the policy and the prediction model. Experimental evaluations on several high-dimensional locomotion tasks show that our learned policies produce more cyclical and consistent action trajectories, and achieve superior performance, and robustness to noise and dynamic changes than the state-of-the-art.