mlsanityData to 7 Oct 2026

Compact Language Models via Pruning and Knowledge Distillation

arXiv 2407.14679 ↗First posted 19 Jul 2024cs.CL cs.AI cs.LGcited by 142

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) targeting different deployment scales and sizes are currently produced by training each variant from scratch; this is extremely compute-intensive. In this paper, we investigate if pruning an existing LLM and then re-training it with a fraction (<3%) of the original training data can be a suitable alternative to repeated, full retraining. To this end, we develop a set of practical and effective compression best practices for LLMs that combine depth, width, attention and MLP pruning with knowledge distillation-based retraining; we arrive at these best practices through a detailed empirical exploration of pruning strategies for each axis, methods to combine axes, distillation strategies, and search techniques for arriving at optimal compressed architectures. We use this guide to compress the Nemotron-4 family of LLMs by a factor of 2-4x, and compare their performance to similarly-sized models on a variety of language modeling tasks. Deriving 8B and 4B models from an already pretrained 15B model using our approach requires up to 40x fewer training tokens per model compared to training from scratch; this results in compute cost savings of 1.8x for training the full model family (15B, 8B, and 4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch, perform comparably to other community models such as Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art compression techniques from the literature. We have open-sourced Minitron model weights on Huggingface, with corresponding supplementary material including example code available on GitHub.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

40 of 50 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2016Layer Normalizationcited by 5,579
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
28 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  12. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  13. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  15. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  16. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  17. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  18. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  19. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  20. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  21. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  22. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  23. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  24. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  25. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  26. 2020A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutionscited by 32
  27. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  28. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 72; 2026: 53.

142 papers.

Sort:

By area:

  1. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  2. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  3. 2026Ministral 3cited by 88
  4. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  5. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  6. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  7. 2025Distillation Scaling Lawscited by 59
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  9. 2025The Curse of Depth in Large Language Modelscited by 58
  10. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  11. 2024Densing Law of LLMscited by 39
  12. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
38 more
  1. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  2. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  3. 2025Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matchingcited by 23
  4. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  5. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  6. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  7. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  8. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  9. 2024Mobile Video Diffusioncited by 17
  10. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  11. 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
  12. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoningcited by 15
  13. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  14. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  15. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  16. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  17. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 12
  18. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  19. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  20. 2024KIND: Knowledge Integration and Diversion for Training Decomposable Modelscited by 10
  21. 2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
  22. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  23. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  24. 2025Independence Tests for Language Modelscited by 10
  25. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 10
  26. 2024Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labelscited by 9
  27. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  28. 2025Normalization in Attention Dynamicscited by 9
  29. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  30. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 8
  31. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  32. 2025Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformerscited by 7
  33. 2026Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillationcited by 7
  34. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 7
  35. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6
  36. 2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
  37. 2024BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Modelscited by 5
  38. 2025BlueLM-2.5-3B Technical Reportcited by 5

50 most cited of 142 All 142 (JSON)