mlsanityData to 14 Oct 2026

A Mathematical Model for Curriculum Learning for Parities

arXiv 2301.13833 ↗First posted 31 Jan 2023cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Curriculum learning (CL) - training using samples that are generated and presented in a meaningful order - was introduced in the machine learning context around a decade ago. While CL has been extensively used and analysed empirically, there has been very little mathematical justification for its advantages. We introduce a CL model for learning the class of k-parities on d bits of a binary string with a neural network trained by stochastic gradient descent (SGD). We show that a wise choice of training examples involving two or more product distributions, allows to reduce significantly the computational cost of learning this class of functions, compared to learning under the uniform distribution. Furthermore, we show that for another class of functions - namely the `Hamming mixtures' - CL strategies involving a bounded number of product distributions are not beneficial.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

24 of 46 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  3. 2021Curriculum Learning: A Surveycited by 259
  4. 2017Automated Curriculum Learning for Neural Networkscited by 257
  5. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  7. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  8. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 114
  9. 2014Learning to Executecited by 106
  10. 2017Failures of Gradient-Based Deep Learningcited by 80
  11. 2020Learning Parities with Neural Networkscited by 78
  12. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
12 more
  1. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  2. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  3. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  4. 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
  5. 2021Curriculum learning for language modelingcited by 32
  6. 2021On the Power of Differentiable Learning versus PAC and SQ Learningcited by 28
  7. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  8. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 20
  9. 2016Multi-Task Curriculum Transfer Deep Learning of Clothing Attributescited by 18
  10. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 18
  11. 2017Curriculum Learning for Multi-Task Classification of Visual Attributescited by 13
  12. 2022An initial alignment between neural network and target is needed for gradient descent to learncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 5; 2026: 7.

19 papers.

Sort:

By area:

  1. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  2. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  3. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 16
  4. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  5. 2024Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networkscited by 9
  6. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  7. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  8. 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
  9. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  10. 2025Transformers Provably Learn Sparse XOR with Polylogarithmic Parameterscited by 3
  11. 2025A statistical physics framework for optimal learningcited by 3
  12. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
7 more
  1. 2026Deep Networks Learn Deep Hierarchical Modelscited by 2
  2. 2023Exploring Parity Challenges in Reinforcement Learning through Curriculum Learning with Noisy Labelscited by 1
  3. 2024Statistical curriculum learning: An elimination algorithm achieving an oracle riskcited by 1
  4. 2026The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficientlycited by 1
  5. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  6. 2026Learning through Internalizationcited by 1
  7. 2026The Power of Power Law: Asymmetry Enables Compositional Reasoningcited by 0