# Hierarchical Quality-Diversity for Online Damage Recovery

arXiv: https://arxiv.org/abs/2204.05726 · HTML: https://mlsanity.com/paper/2204.05726/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 33)

- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors](https://mlsanity.com/paper/1905.11874/) (2019)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [A survey on policy search algorithms for learning robot controllers in a handful of trials](https://mlsanity.com/paper/1807.02303/) (2018)
- [Unsupervised Behaviour Discovery with Quality-Diversity Optimisation](https://mlsanity.com/paper/2106.05648/) (2021)
- [Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencoder](https://mlsanity.com/paper/2102.12463/) (2021)
- [Reset-free Trial-and-Error Learning for Robot Damage Recovery](https://mlsanity.com/paper/1610.04213/) (2016)
- [Unsupervised Learning and Exploration of Reachable Outcome Space](https://mlsanity.com/paper/1909.05508/) (2019)
- [Quality Diversity for Multi-task Optimization](https://mlsanity.com/paper/2003.04407/) (2020)
- [From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion](https://mlsanity.com/paper/2011.11722/) (2020)
- [A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomy](https://mlsanity.com/paper/2012.04283/) (2020)
- [Curiosity Driven Exploration of Learned Disentangled Goal Spaces](https://mlsanity.com/paper/1807.01521/) (2018)
- [Finding Game Levels with the Right Difficulty in a Few Trials through Intelligent Trial-and-Error](https://mlsanity.com/paper/2005.07677/) (2020)
- [Illuminating the Space of Beatable Lode Runner Levels Produced By Various Generative Adversarial Networks](https://mlsanity.com/paper/2101.07868/) (2021)
- [From exploration to control: learning object manipulation skills through novelty search and local adaptation](https://mlsanity.com/paper/1901.00811/) (2019)
- [Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems](https://mlsanity.com/paper/2007.01195/) (2020)
- [Hierarchical Behavioral Repertoires with Unsupervised Descriptors](https://mlsanity.com/paper/1804.07127/) (2018)
- [Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion](https://mlsanity.com/paper/2008.11867/) (2020)
- [Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics](https://mlsanity.com/paper/1907.07029/) (2019)
- [MAP-Elites enables Powerful Stepping Stones and Diversity for Modular Robotics](https://mlsanity.com/paper/2012.04375/) (2020)
- [BR-NS: an Archive-less Approach to Novelty Search](https://mlsanity.com/paper/2104.03936/) (2021)
- [Adaptive CLF-MPC With Application To Quadrupedal Robots](https://mlsanity.com/paper/2112.04536/) (2021)

## Built on by (most cited first, 5 of 5)

- [Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics](https://mlsanity.com/paper/2403.09930/) (2024)
- [Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity](https://mlsanity.com/paper/2210.09918/) (2022)
- [Mix-ME: Quality-Diversity for Multi-Agent Learning](https://mlsanity.com/paper/2311.01829/) (2023)
- [Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains](https://mlsanity.com/paper/2304.03672/) (2023)
- [Quality-Diversity Optimisation on a Physical Robot Through Dynamics-Aware and Reset-Free Learning](https://mlsanity.com/paper/2304.12080/) (2023)

## Abstract (arXiv)

Adaptation capabilities, like damage recovery, are crucial for the deployment of robots in complex environments. Several works have demonstrated that using repertoires of pre-trained skills can enable robots to adapt to unforeseen mechanical damages in a few minutes. These adaptation capabilities are directly linked to the behavioural diversity in the repertoire. The more alternatives the robot has to execute a skill, the better are the chances that it can adapt to a new situation. However, solving complex tasks, like maze navigation, usually requires multiple different skills. Finding a large behavioural diversity for these multiple skills often leads to an intractable exponential growth of the number of required solutions. In this paper, we introduce the Hierarchical Trial and Error algorithm, which uses a hierarchical behavioural repertoire to learn diverse skills and leverages them to make the robot more adaptive to different situations. We show that the hierarchical decomposition of skills enables the robot to learn more complex behaviours while keeping the learning of the repertoire tractable. The experiments with a hexapod robot show that our method solves maze navigation tasks with 20% less actions in the most challenging scenarios than the best baseline while having 57% less complete failures.