mlsanityData to 7 Oct 2026

DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

arXiv 2503.07067 ↗First posted 10 Mar 2025cs.CL cs.AI cs.LGcited by 73

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the success of distillation in large language models (LLMs), most prior work applies identical loss functions to both teacher- and student-generated data. These strategies overlook the synergy between loss formulations and data types, leading to a suboptimal performance boost in student models. To address this, we propose DistiLLM-2, a contrastive approach that simultaneously increases the likelihood of teacher responses and decreases that of student responses by harnessing this synergy. Our extensive experiments show that DistiLLM-2 not only builds high-performing student models across a wide range of tasks, including instruction-following and code generation, but also supports diverse applications, such as preference alignment and vision-language extensions. These findings highlight the potential of a contrastive approach to enhance the efficacy of LLM distillation by effectively aligning teacher and student models across varied data types.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 57 of its 66 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

57 of 66 references.

References · 57

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  13. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  16. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  17. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  18. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  20. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  21. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  22. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  23. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  24. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  25. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  26. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  27. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  28. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  29. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  30. 2024Iterative Reasoning Preference Optimizationcited by 244
  31. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  32. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  33. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  34. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  35. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  36. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  37. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  38. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 59.

73 papers.

Sort:

By area:

  1. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  2. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
  3. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  4. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  5. 2026Trust Region On-Policy Distillationcited by 26
  6. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  7. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 12
  8. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoningcited by 11
  9. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  10. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  11. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  12. 2026DOPD: Dual On-policy Distillationcited by 9
38 more
  1. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  2. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  3. 2025MemLoRA: Distilling Expert Adapters for On-Device Memory Systemscited by 7
  4. 2026Hybrid Policy Distillation for LLMscited by 7
  5. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  6. 2025Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillationcited by 6
  7. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 6
  8. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
  9. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  10. 2026Breaking the Tokenizer Barrier: On-Policy Distillation across Model Familiescited by 5
  11. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  12. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  13. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 5
  14. 2026VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillationcited by 5
  15. 2025Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Frameworkcited by 4
  16. 2025ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillationcited by 4
  17. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  18. 2025EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokenscited by 4
  19. 2026Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridgingcited by 4
  20. 2026Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packingcited by 3
  21. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  22. 2025Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillationcited by 2
  23. 2025AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distributioncited by 2
  24. 2026Memorization Dynamics in Knowledge Distillation for Language Modelscited by 2
  25. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 2
  26. 2026MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillationcited by 2
  27. 2026The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputscited by 2
  28. 2026KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Searchcited by 2
  29. 2026Geometric Self-Distillation for Reasoning Generalizationcited by 2
  30. 2026When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillationcited by 2
  31. 2026Compete Then Collaborate: Frontier AI Teachers Build a Verifiable Curriculum to Improve a Coding Student Beyond Imitationcited by 2
  32. 2026Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Modelscited by 2
  33. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  34. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspectivecited by 1
  35. 2026Validity-Calibrated Reasoning Distillationcited by 1
  36. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Modelscited by 1
  37. 2026When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selectioncited by 1
  38. 2026DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizercited by 1

50 most cited of 73 All 73 (JSON)