mlsanityData to 7 Oct 2026

In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models

arXiv 2212.10670 ↗First posted 20 Dec 2022cs.CL cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Given the success with in-context learning of large pre-trained language models, we introduce in-context learning distillation to transfer in-context few-shot learning ability from large models to smaller models. We propose to combine in-context learning objectives with language modeling objectives to distill both the ability to read in-context examples and task knowledge to the smaller models. We perform in-context learning distillation under two different few-shot learning paradigms: Meta In-context Tuning (Meta-ICT) and Multitask In-context Tuning (Multitask-ICT). Multitask-ICT performs better on multitask few-shot learning but also requires more computation than Meta-ICT. Our method shows consistent improvements for both Meta-ICT and Multitask-ICT on two benchmarks: LAMA and CrossFit. Our extensive experiments and analysis reveal that in-context learning objectives and language modeling objectives are complementary under the Multitask-ICT paradigm. In-context learning objectives achieve the best performance when combined with language modeling objectives.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

34 of 42 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
22 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  2. 2021MetaICL: Learning to Learn In Contextcited by 450
  3. 2021True Few-Shot Learning with Language Modelscited by 343
  4. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  5. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  6. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  7. 2021Meta-learning via Language Model In-context Tuningcited by 163
  8. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  9. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  10. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  11. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  12. 2021Annealing Knowledge Distillationcited by 61
  13. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  14. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  15. 2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
  16. 2019Revisiting Knowledge Distillation via Label Smoothing Regularizationcited by 29
  17. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 25
  18. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  19. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  20. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  21. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  22. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 21; 2025: 7; 2026: 4.

36 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  9. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  10. 2024A Survey on Transformer Compressioncited by 44
  11. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  12. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
24 more
  1. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  2. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  3. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 33
  4. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  5. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  6. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 20
  7. 2025When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Modelscited by 12
  8. 2026Self-Supervised On-Policy Distillation for Reasoning Language Modelscited by 10
  9. 2024PatentGPT: A Large Language Model for Intellectual Propertycited by 8
  10. 2023A Performance Evaluation of a Quantized Large Language Model on Various Smartphonescited by 7
  11. 2026PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasonerscited by 7
  12. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  13. 2025Unlocking the Potential of Generative AI through Neuro-Symbolic Architectures: Benefits and Limitationscited by 6
  14. 2024CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Taskscited by 5
  15. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  16. 2024LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deploymentcited by 4
  17. 2024FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigationcited by 3
  18. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerationscited by 2
  19. 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 2
  20. 2025Improving In-Context Learning with Reasoning Distillationcited by 1
  21. 2024AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformerscited by 0
  22. 2024FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routingcited by 0
  23. 2025Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translationcited by 0
  24. 2026OPSRD: On-Policy Self-Role Distillationcited by 0