mlsanityData to 7 Oct 2026

Student as an Inherent Denoiser of Noisy Teacher

arXiv 2312.10185 ↗First posted 15 Dec 2023cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by teacher models are usually noisy and may influence KD performance. This study delves into KD with noisy teachers and uncovers that the student model can already generate more accurate predictions than the teacher labels used to train it during KD, indicating its inherent ability to denoise noisy teacher labels. Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM by approximately 5% with 50 human-labeled data, and even competitive to standard supervised finetuning with 750 human-labeled data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

28 of 37 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  8. 2016Sequence-Level Knowledge Distillationcited by 761
  9. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  10. 2018Born Again Neural Networkscited by 539
  11. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  12. 2022Large Language Models Are Reasoning Teacherscited by 318
16 more
  1. 2022Is GPT-3 a Good Data Annotator?cited by 208
  2. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  3. 2021Does Knowledge Distillation Really Work?cited by 187
  4. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  5. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  6. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  7. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  8. 2014Grammar as a Foreign Languagecited by 70
  9. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  10. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  11. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  12. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  13. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
  14. 2017Fidelity-Weighted Learningcited by 19
  15. 2020On the Role of Supervision in Unsupervised Constituency Parsingcited by 18
  16. 2022Weighted Distillation with Unlabeled Examplescited by 12

Built on by

Not cited yet.