# Proximal Curriculum for Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2304.12877 · HTML: https://mlsanity.com/paper/2304.12877/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 70 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 30 of 42)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [Learning to Execute](https://mlsanity.com/paper/1410.4615/) (2014)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018)
- [Learning Montezuma's Revenge from a Single Demonstration](https://mlsanity.com/paper/1812.03381/) (2018)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots](https://mlsanity.com/paper/1301.4862/) (2013)
- [Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation](https://mlsanity.com/paper/2010.01114/) (2020)
- [Self-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2004.11812/) (2020)
- [Iterative Machine Teaching](https://mlsanity.com/paper/1705.10470/) (2017)

## Built on by (most cited first, 20 of 20)

- [Self-Evolving Curriculum for LLM Reasoning](https://mlsanity.com/paper/2505.14970/) (2025)
- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery](https://mlsanity.com/paper/2408.15099/) (2024)
- [Learning to Reason at the Frontier of Learnability](https://mlsanity.com/paper/2502.12272/) (2025)
- [An Optimisation Framework for Unsupervised Environment Design](https://mlsanity.com/paper/2505.20659/) (2025)
- [Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training](https://mlsanity.com/paper/2602.20532/) (2026)
- [Proximal Curriculum with Task Correlations for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02481/) (2024)
- [Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR](https://mlsanity.com/paper/2606.25178/) (2026)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [Syllabus: Portable Curricula for Reinforcement Learning Agents](https://mlsanity.com/paper/2411.11318/) (2024)
- [Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs](https://mlsanity.com/paper/2511.02690/) (2025)
- [AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO](https://mlsanity.com/paper/2606.06828/) (2026)
- [TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design](https://mlsanity.com/paper/2506.19997/) (2025)
- [Causal-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2507.02910/) (2025)
- [Improving Regret Approximation for Unsupervised Dynamic Environment Generation](https://mlsanity.com/paper/2601.14957/) (2026)
- [Scaling Curriculum Learning For Autonomous Driving](https://mlsanity.com/paper/2608.22549/) (2026)
- [Learning Embeddings for Sequential Tasks Using Population of Agents](https://mlsanity.com/paper/2306.03311/) (2023)
- [Knowledge capture, adaptation and composition (KCAC): A framework for cross-task curriculum learning in robotic manipulation](https://mlsanity.com/paper/2505.10522/) (2025)
- [LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL](https://mlsanity.com/paper/2607.04412/) (2026)
- [BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation](https://mlsanity.com/paper/2610.13285/) (2026)

## Abstract (arXiv)

We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings. Existing techniques on automatic curriculum design typically require domain-specific hyperparameter tuning or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone of Proximal Development (ZPD). ProCuRL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive ProCuRL by analyzing two simple learning settings. We also present a practical variant of ProCuRL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on a variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.