# Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion

arXiv: https://arxiv.org/abs/2109.06409 · HTML: https://mlsanity.com/paper/2109.06409/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Multi-expert learning of adaptive legged locomotion](https://mlsanity.com/paper/2012.05810/) (2020)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Robust Quadrupedal Locomotion on Sloped Terrains: A Linear Policy Approach](https://mlsanity.com/paper/2010.16342/) (2020)
- [Line Walking and Balancing for Legged Robots with Point Feet](https://mlsanity.com/paper/2007.01087/) (2020)

## Built on by (most cited first, 14 of 14)

- [CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion](https://mlsanity.com/paper/2211.00458/) (2022)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformer](https://mlsanity.com/paper/2212.07740/) (2022)
- [Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner](https://mlsanity.com/paper/2111.00262/) (2021)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [AcL: Action Learner for Fault-Tolerant Quadruped Locomotion Control](https://mlsanity.com/paper/2503.21401/) (2025)
- [An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement](https://mlsanity.com/paper/2403.01962/) (2024)
- [Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation](https://mlsanity.com/paper/2209.12029/) (2022)
- [Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning](https://mlsanity.com/paper/2209.11596/) (2022)
- [A Central Motor System Inspired Pre-training Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2311.07822/) (2023)
- [Locomotion Generation for a Rat Robot based on Environmental Changes via Reinforcement Learning](https://mlsanity.com/paper/2403.11788/) (2024)
- [Energy-Efficient Omnidirectional Locomotion for Wheeled Quadrupeds via Predictive Energy-Aware Nominal Gait Selection](https://mlsanity.com/paper/2601.10723/) (2025)
- [Robust Quadruped Locomotion via Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2604.07224/) (2026)

## Abstract (arXiv)

Recently reinforcement learning (RL) has emerged as a promising approach for quadrupedal locomotion, which can save the manual effort in conventional approaches such as designing skill-specific controllers. However, due to the complex nonlinear dynamics in quadrupedal robots and reward sparsity, it is still difficult for RL to learn effective gaits from scratch, especially in challenging tasks such as walking over the balance beam. To alleviate such difficulty, we propose a novel RL-based approach that contains an evolutionary foot trajectory generator. Unlike prior methods that use a fixed trajectory generator, the generator continually optimizes the shape of the output trajectory for the given task, providing diversified motion priors to guide the policy learning. The policy is trained with reinforcement learning to output residual control signals that fit different gaits. We then optimize the trajectory generator and policy network alternatively to stabilize the training and share the exploratory data to improve sample efficiency. As a result, our approach can solve a range of challenging tasks in simulation by learning from scratch, including walking on a balance beam and crawling through the cave. To further verify the effectiveness of our approach, we deploy the controller learned in the simulation on a 12-DoF quadrupedal robot, and it can successfully traverse challenging scenarios with efficient gaits.