mlsanityData to 14 Oct 2026

DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning

arXiv 2109.07380 ↗First posted 15 Sep 2021cs.LG cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Deep reinforcement learning (RL) has shown great empirical successes, but suffers from brittleness and sample inefficiency. A potential remedy is to use a previously-trained policy as a source of supervision. In this work, we refer to these policies as teachers and study how to transfer their expertise to new student policies by focusing on data usage. We propose a framework, Data CUrriculum for Reinforcement learning (DCUR), which first trains teachers using online deep RL, and stores the logged environment interaction history. Then, students learn by running either offline RL or by using teacher data in combination with a small amount of self-generated data. DCUR's central idea involves defining a class of data curricula which, as a function of training time, limits the student to sampling from a fixed subset of the full teacher data. We test teachers and students using state-of-the-art deep RL algorithms across a variety of data curricula. Results suggest that the choice of data curricula significantly impacts student learning, and that it is beneficial to limit the data during early training stages while gradually letting the data availability grow over time. We identify when the student can learn offline and match teacher performance without relying on specialized offline RL algorithms. Furthermore, we show that collecting a small fraction of online data provides complementary benefits with the data curriculum. Supplementary material is available at https://tinyurl.com/teach-dcur.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 68 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

45 of 68 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  3. 2020Critic Regularized Regression26 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  11. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 34

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2016OpenAI Gymcited by 1,736
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  10. 2015Policy Distillationcited by 440
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
22 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  2. 2017Teacher-Student Curriculum Learningcited by 210
  3. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  4. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  5. 2020Revisiting Fundamentals of Experience Replaycited by 175
  6. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  7. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  8. 2020When Do Curricula Work?cited by 106
  9. 2017A Deeper Look at Experience Replaycited by 105
  10. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  11. 2019Distilling Policy Distillationcited by 84
  12. 2018Kickstarting Deep Reinforcement Learningcited by 81
  13. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  14. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  15. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 44
  16. 2018Learning to Teach with Dynamic Loss Functionscited by 40
  17. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  18. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 36
  19. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 34
  20. 2019AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teacherscited by 28
  21. 2020Dual Policy Distillationcited by 25
  22. 2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Teacher-student curriculum learning for reinforcement learning
  2. 2025Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation
  3. 2024Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning
  4. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
  5. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  6. 2026A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies
  7. 2026What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
  8. 2024Augmenting Offline RL with Unlabeled Data