mlsanityData to 14 Oct 2026

Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation

arXiv 2210.10195 ↗First posted 18 Oct 2022cs.LG cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy ones and gradually learning towards difficult tasks. In this work, we focus on the idea of framing CRL as interpolations between a source (auxiliary) and a target task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT, which formulates CRL as an optimal transport problem with a tailored distance metric between tasks. Specifically, we generate a sequence of task distributions as a geodesic interpolation (i.e., Wasserstein barycenter) between the source and target distributions. Different from many existing methods, our algorithm considers a task-dependent contextual distance metric and is capable of handling nonparametric distributions in both continuous and discrete context settings. In addition, we theoretically show that GRADIENT enables smooth transfer between subsequent stages in the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms of learning efficiency and asymptotic performance.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 42 of its 80 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

42 of 80 references.

References · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017Robust Adversarial Reinforcement Learningcited by 451
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  10. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  11. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  12. 2013Fast Computation of Wasserstein Barycenterscited by 216
30 more
  1. 2019Active Domain Randomizationcited by 170
  2. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  3. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  4. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 119
  5. 2020Continuously Indexed Domain Adaptationcited by 75
  6. 2019Exploration via Hindsight Goal Generationcited by 54
  7. 2021Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learningcited by 53
  8. 2017Incremental Adversarial Domain Adaptation for Continually Changing Environmentscited by 52
  9. 2020Self-Paced Deep Reinforcement Learningcited by 50
  10. 2020Scalable Computations of Wasserstein Barycenter via Input Convex Neural Networkscited by 50
  11. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  12. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  13. 2020Debiased Sinkhorn barycenterscited by 34
  14. 2020Continuous Regularized Wasserstein Barycenterscited by 32
  15. 2022Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyondcited by 32
  16. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  17. 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 28
  18. 2017Learning Wasserstein Embeddingscited by 26
  19. 2019Self-Paced Contextual Reinforcement Learningcited by 26
  20. 2021Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problemscited by 26
  21. 2022Gradual Domain Adaptation without Indexed Intermediate Domainscited by 26
  22. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
  23. 2021A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learningcited by 22
  24. 2021Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absentcited by 21
  25. 2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
  26. 2015Principal Geodesic Analysis for Probability Measures under the Optimal Transport Metriccited by 15
  27. 2018Accuracy-based Curriculum Learning in Deep Reinforcement Learningcited by 15
  28. 2022GeoECG: Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Predictioncited by 8
  29. 2021Functional optimal transport: map estimation and domain adaptation for functional datacited by 7
  30. 2022PhysioMTL: Personalizing Physiological Patterns using Optimal Transport Multi-Task Regressioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 4; 2025: 4; 2026: 7.

28 papers.

Sort:

By area:

  1. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  2. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 20
  3. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 20
  4. 2023Domain Randomization via Entropy Maximizationcited by 19
  5. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 15
  6. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
  7. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  8. 2022Continual Vision-based Reinforcement Learning with Group Symmetriescited by 10
  9. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 10
  10. 2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 7
  11. 2023On the Benefit of Optimal Transport for Curriculum Reinforcement Learningcited by 6
  12. 2025Causally Aligned Curriculum Learningcited by 6
16 more
  1. 2023GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learningcited by 5
  2. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  3. 2023Interpolation for Robust Learning: Data Augmentation on Wasserstein Geodesicscited by 3
  4. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 3
  5. 2023Integrating Curricula with Replays: Its Effects on Continual Learningcited by 3
  6. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  7. 2025Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environmentscited by 1
  8. 2025Causal-Paced Deep Reinforcement Learningcited by 1
  9. 2023Maneuver Decision-Making Through Automatic Curriculum Reinforcement Learning Without Handcrafted Reward functionscited by 0
  10. 2025Autonomous Curriculum Design via Relative Entropy Based Task Modificationscited by 0
  11. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0
  12. 2026Self Paced Gaussian Contextual Reinforcement Learningcited by 0
  13. 2026Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Samplingcited by 0
  14. 2026Curriculum reinforcement learning with measurable task representation learningcited by 0
  15. 2026Optimal Training-Time Scaling in Gradual Adaptationcited by 0
  16. 2026Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesicscited by 0