# Open-Ended Learning Strategies for Learning Complex Locomotion Skills

arXiv: https://arxiv.org/abs/2206.06796 · HTML: https://mlsanity.com/paper/2206.06796/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents](https://mlsanity.com/paper/1712.06560/) (2017)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1909.12324/) (2019)
- [CARL: Controllable Agent with Reinforcement Learning for Quadruped Locomotion](https://mlsanity.com/paper/2005.03288/) (2020)
- [Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods](https://mlsanity.com/paper/2003.11164/) (2020)
- [Maximum Mutation Reinforcement Learning for Scalable Control](https://mlsanity.com/paper/2007.13690/) (2020)

## Built on by (most cited first, 1 of 1)

- [Augmentative Topology Agents For Open-Ended Learning](https://mlsanity.com/paper/2210.11442/) (2022)

## Abstract (arXiv)

Teaching robots to learn diverse locomotion skills under complex three-dimensional environmental settings via Reinforcement Learning (RL) is still challenging. It has been shown that training agents in simple settings before moving them on to complex settings improves the training process, but so far only in the context of relatively simple locomotion skills. In this work, we adapt the Enhanced Paired Open-Ended Trailblazer (ePOET) approach to train more complex agents to walk efficiently on complex three-dimensional terrains. First, to generate more rugged and diverse three-dimensional training terrains with increasing complexity, we extend the Compositional Pattern Producing Networks - Neuroevolution of Augmenting Topologies (CPPN-NEAT) approach and include randomized shapes. Second, we combine ePOET with Soft Actor-Critic off-policy optimization, yielding ePOET-SAC, to ensure that the agent could learn more diverse skills to solve more challenging tasks. Our experimental results show that the newly generated three-dimensional terrains have sufficient diversity and complexity to guide learning, that ePOET successfully learns complex locomotion skills on these terrains, and that our proposed ePOET-SAC approach slightly improves upon ePOET.