# Causal-Paced Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2507.02910 · HTML: https://mlsanity.com/paper/2507.02910/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning](https://mlsanity.com/paper/2010.04296/) (2020)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [A Survey on Causal Discovery: Theory and Practice](https://mlsanity.com/paper/2305.10032/) (2023)
- [Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning](https://mlsanity.com/paper/2010.03110/) (2020)
- [Self-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2004.11812/) (2020)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Self-Paced Contextual Reinforcement Learning](https://mlsanity.com/paper/1910.02826/) (2019)
- [A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning](https://mlsanity.com/paper/2102.13176/) (2021)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum](https://mlsanity.com/paper/2206.10057/) (2022)
- [Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation](https://mlsanity.com/paper/2301.11741/) (2023)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [Causally Aligned Curriculum Learning](https://mlsanity.com/paper/2503.16799/) (2025)
- [Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03429/) (2023)
- [Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement](https://mlsanity.com/paper/2310.19261/) (2023)

## Built on by (most cited first, 1 of 1)

- [When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO](https://mlsanity.com/paper/2610.06861/) (2026)

## Abstract (arXiv)

Designing effective task sequences is crucial for curriculum reinforcement learning (CRL), where agents must gradually acquire skills by training on intermediate tasks. A key challenge in CRL is to identify tasks that promote exploration, yet are similar enough to support effective transfer. While recent approach suggests comparing tasks via their Structural Causal Models (SCMs), the method requires access to ground-truth causal structures, an unrealistic assumption in most RL settings. In this work, we propose Causal-Paced Deep Reinforcement Learning (CP-DRL), a curriculum learning framework aware of SCM differences between tasks based on interaction data approximation. This signal captures task novelty, which we combine with the agent's learnability, measured by reward gain, to form a unified objective. Empirically, CP-DRL outperforms existing curriculum methods on the Point Mass benchmark, achieving faster convergence and higher returns. CP-DRL demonstrates reduced variance with comparable final returns in the Bipedal Walker-Trivial setting, and achieves the highest average performance in the Infeasible variant. These results indicate that leveraging causal relationships between tasks can improve the structure-awareness and sample efficiency of curriculum reinforcement learning. We provide the full implementation of CP-DRL to facilitate the reproduction of our main results at https://github.com/Cho-Geonwoo/CP-DRL.