EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models
On the map: Knowledge distillation → Knowledge distillation for LLMs
Abstract (arXiv)
In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs). Our framework offers versatile functionalities, including data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning techniques specifically tailored for KD scenarios. The toolkit accommodates KD functionalities for both System 1 (fast, intuitive) and System 2 (slow, analytical) models. With its modular design and user-friendly interface, EasyDistill empowers researchers and industry practitioners to seamlessly experiment with and implement state-of-the-art KD strategies for LLMs. In addition, EasyDistill provides a series of robust distilled models and KD-based industrial solutions developed by us, along with the corresponding open-sourced datasets, catering to a variety of use cases. Furthermore, we describe the seamless integration of EasyDistill into Alibaba Cloud's Platform for AI (PAI). Overall, the EasyDistill toolkit makes advanced KD techniques for LLMs more accessible and impactful within the NLP community.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 17 of its 23 references that are in the map. First posted on arXiv 27 May 2025.
Builds on
17 of 23 references.
References · 17
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023A Survey of Large Language Modelscited by 2,576
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
- 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
- 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 119
- 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
5 more
- 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
- 2024Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planningcited by 13
- 2025DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Modelscited by 10
- 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
- 2024Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloudcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024A Survey on Knowledge Distillation of Large Language Models
- 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
- 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models
- 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
- 2024Knowledge Distillation of Black-Box Large Language Models
- 2026KDFP: A first-principles approach to knowledge distillation in large language models
- 2026Hybrid Policy Distillation for LLMs
- 2022AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models