mlsanityData to 14 Oct 2026

Task Diversity Shortens the ICL Plateau

arXiv 2410.05448 ↗First posted 7 Oct 2024cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 73 of its 84 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

73 of 84 references.

References · 73

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  12. 2022Transformers learn in-context by gradient descentcited by 732
38 more
  1. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  3. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  4. 2023Textbooks Are All You Needcited by 569
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  6. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  7. 2021MetaICL: Learning to Learn In Contextcited by 454
  8. 2023Larger language models do in-context learning differentlycited by 394
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  10. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  13. 2023Function Vectors in Large Language Modelscited by 308
  14. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  15. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  16. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  17. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  18. 2015The Benefit of Multitask Representation Learningcited by 195
  19. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  20. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  21. 2020Provable Meta-Learning of Linear Representationscited by 161
  22. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  23. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  24. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  25. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  26. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  27. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  28. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  29. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  30. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  31. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  32. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  33. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  34. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  35. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  36. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  37. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  38. 2023Learning time-scales in two-layers neural networkscited by 57

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  2. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 16
  3. 2025In-Context Algebracited by 5
  4. 2026The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspacescited by 1
  5. 2025In-Context Learning with Hypothesis-Class Guidancecited by 0