# Noisy Self-Knowledge Distillation for Text Summarization

arXiv: https://arxiv.org/abs/2009.07032 · HTML: https://mlsanity.com/paper/2009.07032/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 30 of 31)

- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [Dual-Space Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2406.17328/) (2024)
- [A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training](https://mlsanity.com/paper/2305.02031/) (2023)
- [Attention Temperature Matters in Abstractive Summarization Distillation](https://mlsanity.com/paper/2106.03441/) (2021)
- [Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness](https://mlsanity.com/paper/2205.01889/) (2022)
- [Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation](https://mlsanity.com/paper/2311.08640/) (2023)
- [Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation](https://mlsanity.com/paper/2210.13459/) (2022)
- [Improving the Robustness of Summarization Systems with Dual Augmentation](https://mlsanity.com/paper/2306.01090/) (2023)
- [Multilingual AMR Parsing with Noisy Knowledge Distillation](https://mlsanity.com/paper/2109.15196/) (2021)
- [Enhancing Abstractiveness of Summarization Models through Calibrated Distillation](https://mlsanity.com/paper/2310.13760/) (2023)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [Latent Prompt Tuning for Text Summarization](https://mlsanity.com/paper/2211.01837/) (2022)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs](https://mlsanity.com/paper/2110.10645/) (2021)
- [Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection](https://mlsanity.com/paper/2403.04073/) (2024)
- [Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model](https://mlsanity.com/paper/2210.12427/) (2022)
- [LLM aided semi-supervision for Extractive Dialog Summarization](https://mlsanity.com/paper/2311.11462/) (2023)
- [A Semi-supervised Learning Approach with Two Teachers to Improve Breakdown Identification in Dialogues](https://mlsanity.com/paper/2202.10948/) (2022)
- [DialogSum Challenge: Results of the Dialogue Summarization Shared Task](https://mlsanity.com/paper/2208.03898/) (2022)
- [Reducing Hallucinations in Entity Abstract Summarization with Facts-Template Decomposition](https://mlsanity.com/paper/2402.18873/) (2024)
- [Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentation](https://mlsanity.com/paper/2409.20498/) (2024)
- [CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization](https://mlsanity.com/paper/2503.05750/) (2025)
- [Long document summarization using page specific target text alignment and distilling page importance](https://mlsanity.com/paper/2509.16539/) (2025)
- [Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis](https://mlsanity.com/paper/2310.01209/) (2023)
- [Controlled Randomness Improves the Performance of Transformer Models](https://mlsanity.com/paper/2310.13526/) (2023)
- [Student as an Inherent Denoiser of Noisy Teacher](https://mlsanity.com/paper/2312.10185/) (2023)
- [Leveraging Entailment Judgements in Cross-Lingual Summarisation](https://mlsanity.com/paper/2408.00675/) (2024)
- [Mitigating the Negative Impact of Over-association for Conversational Query Production](https://mlsanity.com/paper/2409.19572/) (2024)
- [Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images](https://mlsanity.com/paper/2604.14506/) (2026)

## Abstract (arXiv)

In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.