mlsanityData to 14 Oct 2026

Enhancing Abstractiveness of Summarization Models through Calibrated Distillation

arXiv 2310.13760 ↗First posted 20 Oct 2023cs.CLcited by 8

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Sequence-level knowledge distillation reduces the size of Seq2Seq models for more efficient abstractive summarization. However, it often leads to a loss of abstractiveness in summarization. In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 47 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
18 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  2. 2016Sequence-Level Knowledge Distillationcited by 801
  3. 2019Text Summarization with Pretrained Encoderscited by 556
  4. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  5. 2013Do Deep Nets Really Need to be Deep?cited by 474
  6. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  7. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  8. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  9. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  10. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  11. 2017Controllable Abstractive Summarizationcited by 110
  12. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  13. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  14. 2020Pre-trained Summarization Distillationcited by 71
  15. 2021Hierarchical Learning for Generation with Long Source Sequencescited by 32
  16. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  17. 2021Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarizationcited by 16
  18. 2021Attention Temperature Matters in Abstractive Summarization Distillationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 2.

8 papers.

  1. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  2. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  3. 2024UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMscited by 21
  4. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languagescited by 11
  5. 2024Factual Dialogue Summarization via Learning from Large Language Modelscited by 6
  6. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  7. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0
  8. 2026Ready2Blend: From Natural-Language Instructions to Composable Alignment Promptscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation
  2. 2026When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
  3. 2024Information-Theoretic Distillation for Reference-less Summarization
  4. 2024Abstractive Text Summarization: State of the Art, Challenges, and Improvements
  5. 2021Mitigating Data Scarceness through Data Synthesis, Augmentation and Curriculum for Abstractive Summarization
  6. 2022Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
  7. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
  8. 2026ScholarSum: Student-Teacher Abstractive Summarization via Knowledge Graph Reasoning and Reflective Refinement