# MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning

arXiv: https://arxiv.org/abs/2403.06914 · HTML: https://mlsanity.com/paper/2403.06914/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://mlsanity.com/paper/2212.06713/) (2022)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [Learning to Generate Task-Specific Adapters from Task Description](https://mlsanity.com/paper/2101.00420/) (2021)

## Built on by (most cited first, 15 of 15)

- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning](https://mlsanity.com/paper/2410.07074/) (2024)
- [Mitigating Copy Bias in In-Context Learning through Neuron Pruning](https://mlsanity.com/paper/2410.01288/) (2024)
- [Generative Prompt Internalization](https://mlsanity.com/paper/2411.15927/) (2024)
- [Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering](https://mlsanity.com/paper/2403.15268/) (2024)
- [DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer](https://mlsanity.com/paper/2507.23554/) (2025)
- [SoftSRV: Learn to Generate Targeted Synthetic Data](https://mlsanity.com/paper/2410.16534/) (2024)
- [On Theoretical Interpretations of Concept-Based In-Context Learning](https://mlsanity.com/paper/2509.20882/) (2025)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [Doc-to-Atom: Learning to Compile and Compose Memory Atoms](https://mlsanity.com/paper/2606.12400/) (2026)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [The Impact of Role Design in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2509.23501/) (2025)
- [Meta-Learning Preferences for Multilingual LLM Alignment](https://mlsanity.com/paper/2607.13315/) (2026)
- [LoRA-generating hypernetworks for efficient on-device LLM generative personalization](https://mlsanity.com/paper/2609.24979/) (2026)

## Abstract (arXiv)

Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LLM's in-context learning performance. To mitigate these challenges, we present Meta dEmonstratioN Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. We exploit the knowledge distillation to enhance alignment between MEND and LLM, achieving both efficiency and effectiveness simultaneously. MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning. Comprehensive evaluations across seven diverse ICL task partitions using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess. It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. This innovation promises enhanced scalability and efficiency for the practical deployment of large language models