mlsanityData to 14 Oct 2026

Learning by Distilling Context

arXiv 2209.15189 ↗First posted 30 Sep 2022cs.CL cs.AIcited by 138

Not on the map. Why?

Cited by 38 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models significantly benefit from context tokens, such as prompts or scratchpads. They perform better when prompted with informative instructions, and they acquire new reasoning capabilities by generating a scratch-pad before predicting the final answers. However, they do not textitinternalize these performance gains, which disappear when the context tokens are gone. Our work proposes to apply context distillation so that a language model can improve itself by internalizing these gains. Concretely, given a synthetic unlabeled input for the target task, we condition the model on ``[instructions] + [task-input]'' to predict ``[scratch-pad] + [final answer]''; then we fine-tune the same model to predict its own ``[final answer]'' conditioned on the ``[task-input]'', without seeing the ``[instructions]'' or using the ``[scratch-pad]''. We show that context distillation is a general method to train language models, and it can effectively internalize 3 types of training signals. First, it can internalize abstract task instructions and explanations, so we can iteratively update the model parameters with new instructions and overwrite old ones. Second, it can internalize step-by-step reasoning for complex tasks (e.g., 8-digit addition), and such a newly acquired capability proves to be useful for other downstream tasks. Finally, it can internalize concrete training examples, and it outperforms directly learning with gradient descent by 9% on the SPIDER Text-to-SQL dataset; furthermore, combining context distillation operations can internalize more training examples than the context window size allows.

Key facts. Cited by 138 papers in the map (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

27 of 38 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
15 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  6. 2021Editing Factual Knowledge in Language Modelscited by 553
  7. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  8. 2021MetaICL: Learning to Learn In Contextcited by 454
  9. 2021Meta-learning via Language Model In-context Tuningcited by 164
  10. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  11. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  12. 2021Towards Zero-Label Language Learningcited by 76
  13. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  14. 2019Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understandingcited by 37
  15. 2022Prompt Injection: Parameterization of Fixed Inputscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 15; 2024: 18; 2025: 18; 2026: 81.

138 papers.

Sort:

By area:

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  4. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2026Reinforcement Learning via Self-Distillationcited by 332
  9. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  11. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  12. 2026Self-Distillation Enables Continual Learningcited by 247
38 more
  1. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  2. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  3. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  4. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  5. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 108
  6. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  7. 2026Privileged Information Distillation for Language Modelscited by 90
  8. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  9. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  10. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 79
  11. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  12. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  13. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  14. 2024Synthetic continued pretrainingcited by 60
  15. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  16. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  17. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  18. 2023Specific versus General Principles for Constitutional AIcited by 47
  19. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  20. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43
  21. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  22. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  23. 2023In-Context Learning with Many Demonstration Examplescited by 39
  24. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  25. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  26. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  27. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  28. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  29. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  30. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  31. 2026Aligning Language Models from User Interactionscited by 28
  32. 2025Antidistillation Samplingcited by 25
  33. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 25
  34. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 24
  35. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 24
  36. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  37. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 19
  38. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  2. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
  3. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation
  4. 2021Causal Distillation for Language Models
  5. 2026On-Policy Context Distillation for Language Models
  6. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
  7. 2021Distilling Linguistic Context for Language Model Compression
  8. 2026Task-Specific Knowledge Distillation via Intermediate Probes