# ELAD: Explanation-Guided Large Language Models Active Distillation

arXiv: https://arxiv.org/abs/2402.13098 · HTML: https://mlsanity.com/paper/2402.13098/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Large Language Model Guided Tree-of-Thought](https://mlsanity.com/paper/2305.08291/) (2023)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019)
- [WT5?! Training Text-to-Text Models to Explain their Predictions](https://mlsanity.com/paper/2004.14546/) (2020)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)

## Built on by (most cited first, 12 of 12)

- [From Selection to Generation: A Survey of LLM-based Active Learning](https://mlsanity.com/paper/2502.11767/) (2025)
- [Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales](https://mlsanity.com/paper/2509.23574/) (2025)
- [GraphNarrator: Generating Textual Explanations for Graph Neural Networks](https://mlsanity.com/paper/2410.15268/) (2024)
- [Skill-Targeted Adaptive Training](https://mlsanity.com/paper/2510.10023/) (2025)
- [Towards Active Synthetic Data Generation for Finetuning Language Models](https://mlsanity.com/paper/2512.00884/) (2025)
- [Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection](https://mlsanity.com/paper/2604.02819/) (2026)
- [Critique-Guided Distillation for Robust Reasoning via Refinement](https://mlsanity.com/paper/2505.11628/) (2025)
- [Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correction](https://mlsanity.com/paper/2502.15261/) (2025)
- [Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models](https://mlsanity.com/paper/2510.24425/) (2025)
- [Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning](https://mlsanity.com/paper/2606.15123/) (2026)
- [Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation](https://mlsanity.com/paper/2609.09702/) (2026)
- [Verb-ICL: Rethinking In-Context Learning for Structured Prediction](https://mlsanity.com/paper/2610.04725/) (2026)

## Abstract (arXiv)

The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve efficient sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model's reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLM knowledge distillation.