mlsanityData to 14 Oct 2026

Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning

arXiv 2505.04193 ↗First posted 7 May 2025cs.LG cs.RO stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive results of deep reinforcement learning in complex control tasks, it is prone to capturing intricate and spurious correlations between observations and actions, leading to failure under slight perturbations to the environment. To tackle this problem, in this work we introduce a novel inductive bias towards simple policies in reinforcement learning. The simplicity inductive bias is introduced by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories. Our reinforcement learning agent, Trajectory Entropy Reinforcement Learning, is optimized to minimize the trajectory entropy while maximizing rewards. We show that the trajectory entropy can be effectively estimated by learning a variational parameterized action prediction model, and use the prediction model to construct an information-regularized reward function. Furthermore, we construct a practical algorithm that enables the joint optimization of models, including the policy and the prediction model. Experimental evaluations on several high-dimensional locomotion tasks show that our learned policies produce more cyclical and consistent action trajectories, and achieve superior performance, and robustness to noise and dynamic changes than the state-of-the-art.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 48 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

31 of 48 references.

References · 31

  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  2. 2018DeepMind Control Suitecited by 956
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  5. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  6. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  7. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  8. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 127
  9. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  10. 2024Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Controlcited by 99
  11. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  12. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
19 more
  1. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  2. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  3. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  4. 2020Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Drivingcited by 34
  5. 2021Robust Predictable Controlcited by 29
  6. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  7. 2023Leveraging Symmetries in Pick and Placecited by 19
  8. 2023Robust Quadrupedal Locomotion via Risk-Averse Policy Learningcited by 19
  9. 2019Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learningcited by 18
  10. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  11. 2021Composable Energy Policies for Reactive Motion Generation and Reinforcement Learningcited by 16
  12. 2022Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Imagescited by 12
  13. 2023Reinforcement Learning with Simple Sequence Priorscited by 11
  14. 2023Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditionscited by 5
  15. 2021Entropy Maximization for Partially Observable Markov Decision Processescited by 4
  16. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  17. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  18. 2020Predictive Modeling of Periodic Behavior for Human-Robot Symbiotic Walkingcited by 2
  19. 2022Entropy Enhanced Multi-Agent Coordination Based on Hierarchical Graph Learning for Continuous Action Spacecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Maximum Total Correlation Reinforcement Learning
  2. 2024Latent Action Priors for Locomotion with Deep Reinforcement Learning
  3. 2025Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning
  4. 2025Trajectory First: A Curriculum for Discovering Diverse Policies
  5. 2026STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
  6. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learning
  7. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
  8. 2026Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning