mlsanityData to 7 Oct 2026

Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

arXiv 2406.01589 ↗First posted 3 Jun 2024stat.ML cond-mat.dis-nn cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

26 of 38 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2021Curriculum Learning: A Surveycited by 257
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  10. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  11. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 113
  12. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
14 more
  1. 2020When Do Curricula Work?cited by 105
  2. 2018Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrievalcited by 71
  3. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  4. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  5. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  6. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  7. 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
  8. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  9. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
  10. 2021Procrustean Training for Imbalanced Deep Learningcited by 24
  11. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  12. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  13. 2020Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrievalcited by 18
  14. 2022Bias-inducing geometries: an exactly solvable data model with fairness implicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  2. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  3. 2025Influence Dynamics and Stagewise Data Attributioncited by 5
  4. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  5. 2025A statistical physics framework for optimal learningcited by 3
  6. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
  7. 2025Growing Neural Networks: Dynamic Evolution through Gradient Descentcited by 1
  8. 2026Level Up: Defining and Exploiting Transitional Problems for Curriculum Learningcited by 0
  9. 2026Pretraining Curricula Enable Selective Fine-tuningcited by 0