mlsanityData to 7 Oct 2026

Noisy Self-Knowledge Distillation for Text Summarization

arXiv 2009.07032 ↗First posted 15 Sep 2020cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 9; 2024: 6; 2025: 2; 2026: 2.

31 papers.

Sort:

  1. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  2. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  3. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  4. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  5. 2021Attention Temperature Matters in Abstractive Summarization Distillationcited by 14
  6. 2022Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awarenesscited by 13
  7. 2023Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generationcited by 10
  8. 2022Adaptive Label Smoothing with Self-Knowledge in Natural Language Generationcited by 9
  9. 2023Improving the Robustness of Summarization Systems with Dual Augmentationcited by 9
  10. 2021Multilingual AMR Parsing with Noisy Knowledge Distillationcited by 8
  11. 2023Enhancing Abstractiveness of Summarization Models through Calibrated Distillationcited by 8
  12. 2021Language Modelling via Learning to Rankcited by 7
19 more
  1. 2022Latent Prompt Tuning for Text Summarizationcited by 6
  2. 2023ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understandingcited by 6
  3. 2021Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNscited by 5
  4. 2024Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selectioncited by 5
  5. 2022Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Modelcited by 4
  6. 2023LLM aided semi-supervision for Extractive Dialog Summarizationcited by 4
  7. 2022A Semi-supervised Learning Approach with Two Teachers to Improve Breakdown Identification in Dialoguescited by 3
  8. 2022DialogSum Challenge: Results of the Dialogue Summarization Shared Taskcited by 3
  9. 2024Reducing Hallucinations in Entity Abstract Summarization with Facts-Template Decompositioncited by 1
  10. 2024Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentationcited by 1
  11. 2025CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarizationcited by 1
  12. 2025Long document summarization using page specific target text alignment and distilling page importancecited by 1
  13. 2023Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysiscited by 0
  14. 2023Controlled Randomness Improves the Performance of Transformer Modelscited by 0
  15. 2023Student as an Inherent Denoiser of Noisy Teachercited by 0
  16. 2024Leveraging Entailment Judgements in Cross-Lingual Summarisationcited by 0
  17. 2024Mitigating the Negative Impact of Over-association for Conversational Query Productioncited by 0
  18. 2026Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical imagescited by 0
  19. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0