# Pre-Training to Learn in Context

arXiv: https://arxiv.org/abs/2305.09137 · HTML: https://mlsanity.com/paper/2305.09137/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 94 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 30 of 66)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)

## Built on by (most cited first, 30 of 42)

- [In-context Pretraining: Language Modeling Beyond Document Boundaries](https://mlsanity.com/paper/2310.10638/) (2023)
- [Inference Scaling for Long-Context Retrieval Augmented Generation](https://mlsanity.com/paper/2410.04343/) (2024)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [Adapting Large Language Models to Domains via Reading Comprehension](https://mlsanity.com/paper/2309.09530/) (2023)
- [Understanding In-Context Learning via Supportive Pretraining Data](https://mlsanity.com/paper/2306.15091/) (2023)
- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [Mixture of In-Context Prompters for Tabular PFNs](https://mlsanity.com/paper/2405.16156/) (2024)
- [MiniPLM: Knowledge Distillation for Pre-Training Language Models](https://mlsanity.com/paper/2410.17215/) (2024)
- [Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls](https://mlsanity.com/paper/2510.01631/) (2025)
- [Analysing The Impact of Sequence Composition on Language Model Pre-Training](https://mlsanity.com/paper/2402.13991/) (2024)
- [RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models](https://mlsanity.com/paper/2508.02062/) (2025)
- [ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training](https://mlsanity.com/paper/2403.07920/) (2024)
- [Supervised Knowledge Makes Large Language Models Better In-context Learners](https://mlsanity.com/paper/2312.15918/) (2023)
- [Structured Packing in LLM Training Improves Long Context Utilization](https://mlsanity.com/paper/2312.17296/) (2023)
- [Revisiting In-context Learning Inference Circuit in Large Language Models](https://mlsanity.com/paper/2410.04468/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning](https://mlsanity.com/paper/2406.08973/) (2024)
- [Large Language Models are Miscalibrated In-Context Learners](https://mlsanity.com/paper/2312.13772/) (2023)
- [FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries](https://mlsanity.com/paper/2407.10810/) (2024)
- [Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models](https://mlsanity.com/paper/2308.01684/) (2023)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [AutoParLLM: GNN-guided Context Generation for Zero-Shot Code Parallelization using LLMs](https://mlsanity.com/paper/2310.04047/) (2023)
- [Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks](https://mlsanity.com/paper/2310.12477/) (2023)
- [Naive Bayes-based Context Extension for Large Language Models](https://mlsanity.com/paper/2403.17552/) (2024)
- [DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer](https://mlsanity.com/paper/2507.23554/) (2025)
- [Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling](https://mlsanity.com/paper/2501.06256/) (2025)
- [Context-Former: Stitching via Latent Conditioned Sequence Modeling](https://mlsanity.com/paper/2401.16452/) (2024)
- [AICL: Action In-Context Learning for Video Diffusion Model](https://mlsanity.com/paper/2403.11535/) (2024)
- [RARe: Retrieval Augmented Retrieval with In-Context Examples](https://mlsanity.com/paper/2410.20088/) (2024)
- [StaICC: Standardized Evaluation for Classification Task in In-context Learning](https://mlsanity.com/paper/2501.15708/) (2025)

## Abstract (arXiv)

In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.