mlsanityData to 14 Oct 2026

On The Power of Curriculum Learning in Training Deep Networks

arXiv 1904.03626 ↗First posted 7 Apr 2019cs.LG stat.MLcited by 286

Not on the map. Why?

Abstract (arXiv)

Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive ``teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.

Key facts. Cited by 286 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 44; 2022: 49; 2023: 47; 2024: 53; 2025: 51; 2026: 42.

286 papers.

Sort:

By area:

  1. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  2. 2021Curriculum Learning: A Surveycited by 259
  3. 2022SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptationcited by 146
  4. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  5. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 133
  6. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
  7. 2021Manipulating SGD with Data Ordering Attackscited by 74
  8. 2021Learning Fast Sample Re-weighting Without Reward Datacited by 69
  9. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 53
  10. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  11. 2021Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Predictioncited by 48
  12. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
38 more
  1. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  2. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  3. 2021A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classificationcited by 36
  4. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  5. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  6. 2022The Effect of Task Ordering in Continual Learningcited by 31
  7. 2023CUDA: Curriculum of Data Augmentation for Long-Tailed Recognitioncited by 30
  8. 2022Making Recommender Systems Forget: Learning and Unlearning for Erasable Recommendationcited by 29
  9. 2023WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpuscited by 29
  10. 2021Multi-Exit Vision Transformer for Dynamic Inferencecited by 28
  11. 2021CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparabilitycited by 28
  12. 2021Bridging Pre-trained Models and Downstream Tasks for Source Code Understandingcited by 28
  13. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  14. 2021Meta-learning with an Adaptive Task Schedulercited by 27
  15. 2022Learning to Solve Travelling Salesman Problem with Hardness-adaptive Curriculumcited by 27
  16. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Modelscited by 27
  17. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 26
  18. 2024Swift Sampler: Efficient Learning of Sampler by 10 Parameterscited by 25
  19. 2022CLNode: Curriculum Learning for Node Classificationcited by 24
  20. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 23
  21. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  22. 2021Medical Image Segmentation with Limited Supervision: A Review of Deep Network Modelscited by 22
  23. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  24. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 22
  25. 2021Learning with Memory-based Virtual Classes for Deep Metric Learningcited by 21
  26. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  27. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  28. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  29. 2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 18
  30. 2021Membership Inference Attacks are Easier on Difficult Problemscited by 17
  31. 2022Improving Monocular Visual Odometry Using Learned Depthcited by 17
  32. 2023How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budgetcited by 17
  33. 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
  34. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgetscited by 17
  35. 2023Curriculum Graph Machine Learning: A Surveycited by 15
  36. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
  37. 2021Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learningcited by 14
  38. 2021Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathologycited by 13

50 most cited of 286 All 286 (JSON)