mlsanityData to 7 Oct 2026

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

arXiv 2204.13509 ↗First posted 28 Apr 2022cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

24 of 31 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
12 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2021MetaICL: Learning to Learn In Contextcited by 450
  6. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  7. 2021Meta-learning via Language Model In-context Tuningcited by 163
  8. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  9. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  10. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  11. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  12. 2020On the importance of pre-training data volume for compact language modelscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 32; 2024: 18; 2025: 13; 2026: 3.

78 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2022A Survey on In-context Learningcited by 717
  4. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  7. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  8. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  9. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  10. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  11. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  12. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
38 more
  1. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  2. 2022Training Trajectories of Language Models Across Scalescited by 80
  3. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  4. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  5. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  6. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  7. 2023Benchmarking Machine Translation with Cultural Awarenesscited by 53
  8. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  9. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  10. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  11. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43
  12. 2023Pre-Training to Learn in Contextcited by 41
  13. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  14. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  15. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
  16. 2024PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generationcited by 36
  17. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  18. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  19. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  20. 2024SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphscited by 31
  21. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  22. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  23. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  24. 2025Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysiscited by 26
  25. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22
  26. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  27. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  28. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  29. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17
  30. 2023Schema-learning and rebinding as mechanisms of in-context learning and emergencecited by 14
  31. 2024VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correctioncited by 14
  32. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  33. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  34. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Modelscited by 12
  35. 2023Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptionscited by 11
  36. 2024Do LLMs Really Adapt to Domains? An Ontology Learning Perspectivecited by 10
  37. 2023Breaking through the learning plateaus of in-context learning in Transformercited by 8
  38. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7

50 most cited of 78 All 78 (JSON)