# Automated curricula through setter-solver interactions

arXiv: https://arxiv.org/abs/1909.12892 · HTML: https://mlsanity.com/paper/1909.12892/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 26 of 26)

- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Situated Dialogue Learning through Procedural Environment Generation](https://mlsanity.com/paper/2110.03262/) (2021)
- [Training a First-Order Theorem Prover from Synthetic Data](https://mlsanity.com/paper/2103.03798/) (2021)
- [Proximal Curriculum with Task Correlations for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02481/) (2024)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [Causally Aligned Curriculum Learning](https://mlsanity.com/paper/2503.16799/) (2025)
- [Curricular Subgoals for Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.08232/) (2023)
- [Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments](https://mlsanity.com/paper/2207.09597/) (2022)
- [Transferable Curricula through Difficulty Conditioned Generators](https://mlsanity.com/paper/2306.13028/) (2023)
- [Goal-constrained Sparse Reinforcement Learning for End-to-End Driving](https://mlsanity.com/paper/2103.09189/) (2021)
- [Towards Teachable Autotelic Agents](https://mlsanity.com/paper/2105.11977/) (2021)
- [CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2304.03535/) (2023)
- [Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning](https://mlsanity.com/paper/2310.04796/) (2023)
- [Syllabus: Portable Curricula for Reinforcement Learning Agents](https://mlsanity.com/paper/2411.11318/) (2024)
- [Reward-Machine-Guided, Self-Paced Reinforcement Learning](https://mlsanity.com/paper/2305.16505/) (2023)
- [Words as Beacons: Guiding RL Agents with High-Level Language Prompts](https://mlsanity.com/paper/2410.08632/) (2024)
- [Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning](https://mlsanity.com/paper/2209.11596/) (2022)
- [Efficient Mitigation of Bus Bunching through Setter-Based Curriculum Learning](https://mlsanity.com/paper/2405.15824/) (2024)
- [CortexCompile: Harnessing Cortical-Inspired Architectures for Enhanced Multi-Agent NLP Code Synthesis](https://mlsanity.com/paper/2409.02938/) (2024)
- [TEACH: Temporal Variance-Driven Curriculum for Reinforcement Learning](https://mlsanity.com/paper/2512.22824/) (2025)
- [Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals](https://mlsanity.com/paper/2601.19810/) (2026)

## Abstract (arXiv)

Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula--the breakdown of tasks into simpler, static challenges with dense rewards--to build up to complex behaviors. While curricula are also useful for artificial agents, hand-crafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.