# Self-Paced Contextual Reinforcement Learning

arXiv: https://arxiv.org/abs/1910.02826 · HTML: https://mlsanity.com/paper/1910.02826/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 26 of 26)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2011.01734/) (2020)
- [Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems](https://mlsanity.com/paper/2111.04613/) (2021)
- [Specializing Versatile Skill Libraries using Local Mixture of Experts](https://mlsanity.com/paper/2112.04216/) (2021)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [Domain Randomization via Entropy Maximization](https://mlsanity.com/paper/2311.01885/) (2023)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [Hierarchies of Planning and Reinforcement Learning for Robot Navigation](https://mlsanity.com/paper/2109.11178/) (2021)
- [Learning to Learn: How to Continuously Teach Humans and Machines](https://mlsanity.com/paper/2211.15470/) (2022)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [Learning Progress Driven Multi-Agent Curriculum](https://mlsanity.com/paper/2205.10016/) (2022)
- [On Bits and Bandits: Quantifying the Regret-Information Trade-off](https://mlsanity.com/paper/2405.16581/) (2024)
- [Reward-Machine-Guided, Self-Paced Reinforcement Learning](https://mlsanity.com/paper/2305.16505/) (2023)
- [Causal-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2507.02910/) (2025)
- [Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design](https://mlsanity.com/paper/2608.10299/) (2026)
- [Balanced Self-Paced Learning for AUC Maximization](https://mlsanity.com/paper/2207.03650/) (2022)
- [Self-Paced Absolute Learning Progress as a Regularized Approach to Curriculum Learning](https://mlsanity.com/paper/2306.05769/) (2023)
- [Trajectory First: A Curriculum for Discovering Diverse Policies](https://mlsanity.com/paper/2506.01568/) (2025)
- [Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization](https://mlsanity.com/paper/2507.19437/) (2025)
- [Self Paced Gaussian Contextual Reinforcement Learning](https://mlsanity.com/paper/2603.23755/) (2026)
- [Curriculum reinforcement learning with measurable task representation learning](https://mlsanity.com/paper/2605.23372/) (2026)

## Abstract (arXiv)

Generalization and adaptation of learned skills to novel situations is a core requirement for intelligent autonomous robots. Although contextual reinforcement learning provides a principled framework for learning and generalization of behaviors across related tasks, it generally relies on uninformed sampling of environments from an unknown, uncontrolled context distribution, thus missing the benefits of structured, sequential learning. We introduce a novel relative entropy reinforcement learning algorithm that gives the agent the freedom to control the intermediate task distribution, allowing for its gradual progression towards the target context distribution. Empirical evaluation shows that the proposed curriculum learning scheme drastically improves sample efficiency and enables learning in scenarios with both broad and sharp target context distributions in which classical approaches perform sub-optimally.