# Theory of Curriculum Learning, with Convex Loss Functions

arXiv: https://arxiv.org/abs/1812.03472 · HTML: https://mlsanity.com/paper/1812.03472/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2018.

## Built on by (most cited first, 30 of 33)

- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [Data Pruning via Moving-one-Sample-out](https://mlsanity.com/paper/2310.14664/) (2023)
- [Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching](https://mlsanity.com/paper/2402.05011/) (2024)
- [An Analytical Theory of Curriculum Learning in Teacher-Student Networks](https://mlsanity.com/paper/2106.08068/) (2021)
- [Denoising Task Difficulty-based Curriculum for Training Diffusion Models](https://mlsanity.com/paper/2403.10348/) (2024)
- [CLNode: Curriculum Learning for Node Classification](https://mlsanity.com/paper/2206.07258/) (2022)
- [Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models](https://mlsanity.com/paper/2103.00429/) (2021)
- [Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs](https://mlsanity.com/paper/2306.16921/) (2023)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [A Mathematical Model for Curriculum Learning for Parities](https://mlsanity.com/paper/2301.13833/) (2023)
- [Instruction Tuning with Human Curriculum](https://mlsanity.com/paper/2310.09518/) (2023)
- [Optimal Protocols for Continual Learning via Statistical Physics and Control Theory](https://mlsanity.com/paper/2409.18061/) (2024)
- [Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks](https://mlsanity.com/paper/2105.05553/) (2021)
- [On the Statistical Benefits of Curriculum Learning](https://mlsanity.com/paper/2111.07126/) (2021)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks](https://mlsanity.com/paper/2406.01589/) (2024)
- [HiCu: Leveraging Hierarchy for Curriculum Learning in Automated ICD Coding](https://mlsanity.com/paper/2208.02301/) (2022)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training](https://mlsanity.com/paper/2511.07372/) (2025)
- [The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labels](https://mlsanity.com/paper/2210.00583/) (2022)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Task Phasing: Automated Curriculum Learning from Demonstrations](https://mlsanity.com/paper/2210.10999/) (2022)
- [A Hierarchical Assessment of Adversarial Severity](https://mlsanity.com/paper/2108.11785/) (2021)
- [Statistical curriculum learning: An elimination algorithm achieving an oracle risk](https://mlsanity.com/paper/2402.13366/) (2024)
- [TED: Accelerate Model Training by Internal Generalization](https://mlsanity.com/paper/2405.03228/) (2024)
- [Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling](https://mlsanity.com/paper/2411.17567/) (2024)
- [Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning](https://mlsanity.com/paper/2507.12750/) (2025)
- [Interaction-limited Inverse Reinforcement Learning](https://mlsanity.com/paper/2007.00425/) (2020)
- [Deep Learning Training Procedure Augmentations](https://mlsanity.com/paper/2211.14395/) (2022)

## Abstract (arXiv)

Curriculum Learning - the idea of teaching by gradually exposing the learner to examples in a meaningful order, from easy to hard, has been investigated in the context of machine learning long ago. Although methods based on this concept have been empirically shown to improve performance of several learning algorithms, no theoretical analysis has been provided even for simple cases. To address this shortfall, we start by formulating an ideal definition of difficulty score - the loss of the optimal hypothesis at a given datapoint. We analyze the possible contribution of curriculum learning based on this score in two convex problems - linear regression, and binary classification by hinge loss minimization. We show that in both cases, the expected convergence rate decreases monotonically with the ideal difficulty score, in accordance with earlier empirical results. We also prove that when the ideal difficulty score is fixed, the convergence rate is monotonically increasing with respect to the loss of the current hypothesis at each point. We discuss how these results bring to term two apparently contradicting heuristics: curriculum learning on the one hand, and hard data mining on the other.