# Enhancing Abstractiveness of Summarization Models through Calibrated Distillation

arXiv: https://arxiv.org/abs/2310.13760 · HTML: https://mlsanity.com/paper/2310.13760/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Pre-trained Summarization Distillation](https://mlsanity.com/paper/2010.13002/) (2020)
- [Hierarchical Learning for Generation with Long Source Sequences](https://mlsanity.com/paper/2104.07545/) (2021)
- [Noisy Self-Knowledge Distillation for Text Summarization](https://mlsanity.com/paper/2009.07032/) (2020)
- [Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2108.02859/) (2021)
- [Attention Temperature Matters in Abstractive Summarization Distillation](https://mlsanity.com/paper/2106.03441/) (2021)

## Built on by (most cited first, 8 of 8)

- [FineSurE: Fine-grained Summarization Evaluation using LLMs](https://mlsanity.com/paper/2407.00908/) (2024)
- [Learning to Summarize from LLM-generated Feedback](https://mlsanity.com/paper/2410.13116/) (2024)
- [UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs](https://mlsanity.com/paper/2409.19898/) (2024)
- [Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages](https://mlsanity.com/paper/2506.00549/) (2025)
- [Factual Dialogue Summarization via Learning from Large Language Models](https://mlsanity.com/paper/2406.14709/) (2024)
- [References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation](https://mlsanity.com/paper/2505.06552/) (2025)
- [Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data](https://mlsanity.com/paper/2608.23391/) (2026)
- [Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts](https://mlsanity.com/paper/2609.39365/) (2026)

## Abstract (arXiv)

Sequence-level knowledge distillation reduces the size of Seq2Seq models for more efficient abstractive summarization. However, it often leads to a loss of abstractiveness in summarization. In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.