mlsanityData to 14 Oct 2026

Pre-Training to Learn in Context

arXiv 2305.09137 ↗First posted 16 May 2023cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 66 of its 94 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

66 of 94 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020Supervised Contrastive Learningcited by 2,829
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  13. 2022In-context Learning and Induction Headscited by 964
  14. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  15. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  16. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  17. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  18. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  19. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  20. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  21. 2021MetaICL: Learning to Learn In Contextcited by 454
  22. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  23. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  24. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 363
  25. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  26. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  27. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  28. 2019Abductive Commonsense Reasoningcited by 301
  29. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  30. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  31. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  32. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  33. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  34. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  35. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  36. 2021Meta-learning via Language Model In-context Tuningcited by 164
  37. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  38. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 17; 2025: 9; 2026: 6.

42 papers.

Sort:

By area:

  1. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  2. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  3. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  4. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  5. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  6. 2024Data Selection via Optimal Control for Language Modelscited by 38
  7. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  8. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  9. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  10. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 25
  11. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 22
  12. 2024ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Trainingcited by 21
30 more
  1. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  2. 2023Structured Packing in LLM Training Improves Long Context Utilizationcited by 17
  3. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 17
  4. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 16
  5. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  6. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12
  7. 2024FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queriescited by 12
  8. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Modelscited by 9
  9. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  10. 2023AutoParLLM: GNN-guided Context Generation for Zero-Shot Code Parallelization using LLMscited by 5
  11. 2023Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Taskscited by 5
  12. 2024Naive Bayes-based Context Extension for Large Language Modelscited by 5
  13. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  14. 2025Unlocking In-Context Learning for Natural Datasets Beyond Language Modellingcited by 4
  15. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  16. 2024AICL: Action In-Context Learning for Video Diffusion Modelcited by 2
  17. 2024RARe: Retrieval Augmented Retrieval with In-Context Examplescited by 2
  18. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  19. 2025MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learningcited by 2
  20. 2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learningcited by 2
  21. 2023Simple and Effective Input Reformulations for Translationcited by 1
  22. 2024CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Searchcited by 1
  23. 2026Learning Nonlinear Systems In-Context: From Synthetic Data to Real-World Motor Controlcited by 1
  24. 2026In-context Pre-trained Time-Series Foundation Models adapt to Unseen Taskscited by 1
  25. 2025Mechanistic Fine-tuning for In-context Learningcited by 0
  26. 2025Basic Reading Distillationcited by 0
  27. 2026In-context Region-based Drag: Drag Any Region to Any Shapecited by 0
  28. 2026When Context Misleads: In-context Learning with Jurisdiction in Large Language Modelscited by 0
  29. 2026Exploring In-Context Learning for Handwritten Text Recognitioncited by 0
  30. 2026Verb-ICL: Rethinking In-Context Learning for Structured Predictioncited by 0