mlsanityData to 14 Oct 2026

In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

arXiv 2412.13243 ↗First posted 17 Dec 2024cs.CLcited by 4

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 6 of its 6 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

6 of 6 references.

References · 6

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2022Learning by Distilling Contextcited by 138
  5. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  6. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  2. 2026Sample-Efficient Learning from Agent Experiencecited by 3
  3. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
  4. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient LLM Context Distillation
  2. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
  3. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
  4. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation
  5. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  6. 2026Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
  7. 2025Mechanistic Fine-tuning for In-context Learning
  8. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation