# Automaton-Guided Curriculum Generation for Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2304.05271 · HTML: https://mlsanity.com/paper/2304.05271/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [panda-gym: Open-source goal-conditioned environments for robotic learning](https://mlsanity.com/paper/2106.13687/) (2021)
- [An Optimization Framework for Task Sequencing in Curriculum Learning](https://mlsanity.com/paper/1901.11478/) (2019)
- [ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments](https://mlsanity.com/paper/2204.04823/) (2022)
- [A Taxonomy of Similarity Metrics for Markov Decision Processes](https://mlsanity.com/paper/2103.04706/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloning](https://mlsanity.com/paper/2310.08836/) (2023)
- [Reward Inflation: A Healthy Stimulus for Reinforcement Learning](https://mlsanity.com/paper/2610.02545/) (2026)

## Abstract (arXiv)

Despite advances in Reinforcement Learning, many sequential decision making tasks remain prohibitively expensive and impractical to learn. Recently, approaches that automatically generate reward functions from logical task specifications have been proposed to mitigate this issue; however, they scale poorly on long-horizon tasks (i.e., tasks where the agent needs to perform a series of correct actions to reach the goal state, considering future transitions while choosing an action). Employing a curriculum (a sequence of increasingly complex tasks) further improves the learning speed of the agent by sequencing intermediate tasks suited to the learning capacity of the agent. However, generating curricula from the logical specification still remains an unsolved problem. To this end, we propose AGCL, Automaton-guided Curriculum Learning, a novel method for automatically generating curricula for the target task in the form of Directed Acyclic Graphs (DAGs). AGCL encodes the specification in the form of a deterministic finite automaton (DFA), and then uses the DFA along with the Object-Oriented MDP (OOMDP) representation to generate a curriculum as a DAG, where the vertices correspond to tasks, and edges correspond to the direction of knowledge transfer. Experiments in gridworld and physics-based simulated robotics domains show that the curricula produced by AGCL achieve improved time-to-threshold performance on a complex sequential decision-making problem relative to state-of-the-art curriculum learning (e.g, teacher-student, self-play) and automaton-guided reinforcement learning baselines (e.g, Q-Learning for Reward Machines). Further, we demonstrate that AGCL performs well even in the presence of noise in the task's OOMDP description, and also when distractor objects are present that are not modeled in the logical specification of the tasks' objectives.