mlsanityData to 14 Oct 2026

Curriculum learning for language modeling

arXiv 2108.02170 ↗First posted 4 Aug 2021cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Language Models like ELMo and BERT have provided robust representations of natural language, which serve as the language understanding component for a diverse range of downstream tasks.Curriculum learning is a method that employs a structured training regime instead, which has been leveraged in computer vision and machine translation to improve model training speed and model performance. While language models have proven transformational for the natural language processing community, these models have proven expensive, energy-intensive, and challenging to train. In this work, we explore the effect of curriculum learning on language model pretraining using various linguistically motivated curricula and evaluate transfer performance on the GLUE Benchmark. Despite a broad variety of training methodologies and experiments we do not find compelling evidence that curriculum learning methods improve language model training.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

16 of 20 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  10. 2021Curriculum Learning: A Surveycited by 259
  11. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  12. 2021How to Train BERT with an Academic Budgetcited by 80
4 more
  1. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  2. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  3. 2020Optimal Subarchitecture Extraction For BERTcited by 9
  4. 2019Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Rankingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 5; 2024: 8; 2025: 11; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  3. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  4. 2022Prompt Injection: Parameterization of Fixed Inputscited by 36
  5. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  6. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  7. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  8. 2024Curriculum Learning for Small Code Language Modelscited by 22
  9. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20
  10. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  11. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  12. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
20 more
  1. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routingcited by 13
  2. 2022Efficient Pre-training of Masked Language Model via Concept-based Curriculum Maskingcited by 12
  3. 2022Data Selection Curriculum for Neural Machine Translationcited by 11
  4. 2025Curriculum Learning-Driven PIELMs for Fluid Flow Simulationscited by 8
  5. 2025Data Efficacy for Language Model Trainingcited by 8
  6. 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
  7. 2025Investigating the Zone of Proximal Development of Language Models for In-Context Learningcited by 6
  8. 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
  9. 2023On the effect of curriculum learning with developmental data for grammar acquisitioncited by 5
  10. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  11. 2024Interpretability of Language Models via Task Spacescited by 4
  12. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Modelscited by 4
  13. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  14. 2023Curriculum Learning with Adam: The Devil Is in the Wrong Detailscited by 3
  15. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?cited by 3
  16. 2024Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Trainingcited by 3
  17. 2025Scaling LLM Pre-training with Vocabulary Curriculumcited by 3
  18. 2022Development and Comparison of Scoring Functions in Curriculum Learningcited by 1
  19. 2025Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2cited by 1
  20. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0