mlsanityData to 7 Oct 2026

LLMR: Knowledge Distillation with a Large Language Model-Induced Reward

arXiv 2409.12500 ↗First posted 19 Sep 2024cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models have become increasingly popular and demonstrated remarkable performance in various natural language processing (NLP) tasks. However, these models are typically computationally expensive and difficult to be deployed in resource-constrained environments. In this paper, we propose LLMR, a novel knowledge distillation (KD) method based on a reward function induced from large language models. We conducted experiments on multiple datasets in the dialogue generation and summarization tasks. Empirical results demonstrate that our LLMR approach consistently outperforms traditional KD methods in different tasks and datasets.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2016Sequence-Level Knowledge Distillationcited by 761
28 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  2. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  3. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  5. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  6. 2018Model compression via distillation and quantizationcited by 287
  7. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  8. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  10. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  11. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  12. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  13. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  14. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  15. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  16. 2020Pre-trained Summarization Distillationcited by 70
  17. 2020Text Generation by Learning from Demonstrationscited by 64
  18. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  19. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  20. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  21. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  22. 2020Pretrained Language Models for Dialogue Generation with Multiple Input Sourcescited by 14
  23. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12
  24. 2017Why Do Neural Dialog Systems Generate Short and Meaningless Replies? A Comparison between Dialog and Translationcited by 12
  25. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  26. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  27. 2022An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasetscited by 8
  28. 2024EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5
  2. 2025KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillationcited by 2
  3. 2026Cliff: Learning Process Rewards from the First Mistakecited by 1