mlsanityData to 7 Oct 2026

ELAD: Explanation-Guided Large Language Models Active Distillation

arXiv 2402.13098 ↗First posted 20 Feb 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve efficient sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model's reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLM knowledge distillation.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

38 of 57 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
26 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  4. 2020A Survey of Deep Active Learningcited by 547
  5. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  6. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  7. 2022Large Language Models Are Reasoning Teacherscited by 318
  8. 2016Rationalizing Neural Predictionscited by 281
  9. 2022Teaching Small Language Models to Reasoncited by 279
  10. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  11. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  12. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  13. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  14. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  15. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  16. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  17. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  18. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  19. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  20. 2019Rethinking Cooperative Rationalization: Introspective Extraction and Complement Controlcited by 96
  21. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  22. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  23. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  24. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10
  25. 2023Visual Attention Prompted Prediction and Learningcited by 2
  26. 2022KNIFE: Distilling Reasoning Knowledge From Free-Text Rationalescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 3.

11 papers.

  1. 2025From Selection to Generation: A Survey of LLM-based Active Learningcited by 18
  2. 2024GraphNarrator: Generating Textual Explanations for Graph Neural Networkscited by 7
  3. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationalescited by 6
  4. 2025Skill-Targeted Adaptive Trainingcited by 6
  5. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  6. 2026Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selectioncited by 3
  7. 2025Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correctioncited by 1
  8. 2025Critique-Guided Distillation for Robust Reasoning via Refinementcited by 0
  9. 2025Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Modelscited by 0
  10. 2026Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuningcited by 0
  11. 2026Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillationcited by 0