# A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning

arXiv: https://arxiv.org/abs/2102.13176 · HTML: https://mlsanity.com/paper/2102.13176/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 85 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem](https://mlsanity.com/paper/1802.08089/) (2018)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Safe Controller Optimization for Quadrotors with Gaussian Processes](https://mlsanity.com/paper/1509.01066/) (2015)
- [Probabilistic Numerics and Uncertainty in Computations](https://mlsanity.com/paper/1506.01326/) (2015)
- [Markov Decision Processes with Continuous Side Information](https://mlsanity.com/paper/1711.05726/) (2017)
- [Learning Curriculum Policies for Reinforcement Learning](https://mlsanity.com/paper/1812.00285/) (2018)
- [VIREL: A Variational Inference Framework for Reinforcement Learning](https://mlsanity.com/paper/1811.01132/) (2018)
- [Variational Tempering](https://mlsanity.com/paper/1411.1810/) (2014)
- [Accuracy-based Curriculum Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/1806.09614/) (2018)
- [High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewards](https://mlsanity.com/paper/2010.13483/) (2020)
- [Receding Horizon Curiosity](https://mlsanity.com/paper/1910.03620/) (2019)
- [Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning](https://mlsanity.com/paper/1711.11511/) (2017)

## Built on by (most cited first, 22 of 22)

- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula](https://mlsanity.com/paper/2311.01642/) (2023)
- [Proximal Curriculum with Task Correlations for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02481/) (2024)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [Flow-based Domain Randomization for Learning and Sequencing Robotic Skills](https://mlsanity.com/paper/2502.01800/) (2025)
- [Learning Progress Driven Multi-Agent Curriculum](https://mlsanity.com/paper/2205.10016/) (2022)
- [LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World](https://mlsanity.com/paper/2409.19617/) (2024)
- [Probabilistic Curriculum Learning for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2504.01459/) (2025)
- [Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs](https://mlsanity.com/paper/2511.02690/) (2025)
- [Reward-Machine-Guided, Self-Paced Reinforcement Learning](https://mlsanity.com/paper/2305.16505/) (2023)
- [Causal-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2507.02910/) (2025)
- [Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation](https://mlsanity.com/paper/2605.17807/) (2026)
- [Active Curriculum Refinement for Reinforcement Learning](https://mlsanity.com/paper/2608.26469/) (2026)
- [Differentiated Relevances Embedding for Group-based Referring Expression Comprehension](https://mlsanity.com/paper/2203.06382/) (2022)
- [Self-Paced Absolute Learning Progress as a Regularized Approach to Curriculum Learning](https://mlsanity.com/paper/2306.05769/) (2023)
- [Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning](https://mlsanity.com/paper/2503.11467/) (2025)
- [TEACH: Temporal Variance-Driven Curriculum for Reinforcement Learning](https://mlsanity.com/paper/2512.22824/) (2025)
- [Curriculum reinforcement learning with measurable task representation learning](https://mlsanity.com/paper/2605.23372/) (2026)
- [Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulation](https://mlsanity.com/paper/2606.09236/) (2026)

## Abstract (arXiv)

Across machine learning, the use of curricula has shown strong empirical potential to improve learning from data by avoiding local optima of training objectives. For reinforcement learning (RL), curricula are especially interesting, as the underlying optimization has a strong tendency to get stuck in local optima due to the exploration-exploitation trade-off. Recently, a number of approaches for an automatic generation of curricula for RL have been shown to increase performance while requiring less expert knowledge compared to manually designed curricula. However, these approaches are seldomly investigated from a theoretical perspective, preventing a deeper understanding of their mechanics. In this paper, we present an approach for automated curriculum generation in RL with a clear theoretical underpinning. More precisely, we formalize the well-known self-paced learning paradigm as inducing a distribution over training tasks, which trades off between task complexity and the objective to match a desired task distribution. Experiments show that training on this induced distribution helps to avoid poor local optima across RL algorithms in different tasks with uninformative rewards and challenging exploration requirements.