mlsanityData to 14 Oct 2026

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models

arXiv 2411.06839 ↗First posted 11 Nov 2024cs.CL cs.AI cs.LGcited by 4

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstrate that they follow the same paradigm. Inspired by this observation, we propose a parameter-efficient knowledge distillation method, LLM-NEO, which integrates LoRA into KD to improve the efficiency of knowledge transfer. After that, we summarize some valuable guidelines for the hyperparameters in LLM-NEO. Experimental results on compressing Llama 2 and Llama 3.2 show that LLM-NEO outperforms various baselines. Further analysis demonstrates the robustness of the proposed LLM-NEO on variants of LoRA. The code and trained models are available at [Github](https://github.com/yang3121099/LLM-Neo).

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 33 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

26 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models3 Apr

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
13 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  2. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  8. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  11. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  12. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 61
  13. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 37
  2. 2025Shadow-FT: Tuning Instruct Model via Training on Paired Base Modelcited by 10
  3. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  4. 2026Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  2. 2024A Survey on Knowledge Distillation of Large Language Models
  3. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  4. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
  5. 2024Dual-Space Knowledge Distillation for Large Language Models
  6. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
  7. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models
  8. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Reward