mlsanityData to 7 Oct 2026

LLM Pruning and Distillation in Practice: The Minitron Approach

arXiv 2408.11796 ↗First posted 21 Aug 2024cs.CL cs.AI cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo 12B. We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset. We open-source our base model weights on Hugging Face with a permissive license.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

21 of 34 references.

References · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2016Layer Normalizationcited by 5,579
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
9 more
  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  4. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  5. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  6. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  7. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  8. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  9. 2024A deeper look at depth pruning of LLMscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 45; 2026: 24.

82 papers.

Sort:

By area:

  1. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  2. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  3. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  4. 2026Ministral 3cited by 88
  5. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  6. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  7. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  8. 2025Distillation Scaling Lawscited by 59
  9. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  10. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  11. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  12. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
38 more
  1. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  2. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 25
  3. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  4. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  5. 2025DrugPilot: LLM-based Parameterized Reasoning Agent for Drug Discoverycited by 18
  6. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  7. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  8. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  9. 2025CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIscited by 14
  10. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  11. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  12. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  13. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  14. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  15. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  16. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  17. 2025Neodragon: Mobile Video Generation using Diffusion Transformercited by 10
  18. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Modelscited by 8
  19. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  20. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  21. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 7
  22. 2025AI Governance through Marketscited by 5
  23. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  24. 2025Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensationcited by 5
  25. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4
  26. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillationcited by 4
  27. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  28. 2025Iterative Layer Pruning for Efficient Translation Inferencecited by 4
  29. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Modelscited by 4
  30. 2025Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchcited by 3
  31. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMscited by 3
  32. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  33. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Modelscited by 3
  34. 2024BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inferencecited by 2
  35. 2025OverFill: Two-Stage Models for Efficient Language Model Decodingcited by 2
  36. 2025Boomerang Distillation Enables Zero-Shot Model Size Interpolationcited by 2
  37. 2026Energy-Efficient Information Representation in MNIST Classification Using Biologically Inspired Learningcited by 2
  38. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiencycited by 2

50 most cited of 82 All 82 (JSON)