# Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization

arXiv: https://arxiv.org/abs/2108.11846 · HTML: https://mlsanity.com/paper/2108.11846/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 18 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT](https://mlsanity.com/paper/2011.09739/) (2020)

## Built on by (most cited first, 14 of 14)

- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Aligning Language Models with Human Preferences via a Bayesian Approach](https://mlsanity.com/paper/2310.05782/) (2023)
- [On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study](https://mlsanity.com/paper/2304.08653/) (2023)
- [Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets](https://mlsanity.com/paper/2310.13340/) (2023)
- [R2D2: Robust Data-to-Text with Replacement Detection](https://mlsanity.com/paper/2205.12467/) (2022)
- [COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization](https://mlsanity.com/paper/2209.14569/) (2022)
- [Towards Summary Candidates Fusion](https://mlsanity.com/paper/2210.08779/) (2022)
- [FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization](https://mlsanity.com/paper/2303.08335/) (2023)
- [Mitigating Unhelpfulness in Emotional Support Conversations with Multifaceted AI Feedback](https://mlsanity.com/paper/2401.05928/) (2024)
- [Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios](https://mlsanity.com/paper/2603.01288/) (2026)

## Abstract (arXiv)

Encoder-decoder models have achieved remarkable success in abstractive text summarization, which aims to compress one or more documents into a shorter version without the loss of the essential content. Unfortunately, these models mostly suffer a discrepancy between training and inference, i.e., the exposure bias problem. During the training stage, with teacher forcing these models are optimized to maximize the likelihood of the gold summary given the gold summary tokens as input to the decoder, while at inference the given tokens are replaced by the generated tokens. Consequently, low-quality summaries are very likely to be generated. To remedy this problem, we propose to leverage contrastive learning to decrease the likelihood of these low-quality summaries, and meanwhile increase the likelihood of the gold summary. Since our solution expands the states that the model perceives during training, we expect that the exposure bias problem can be alleviated. We experimentally demonstrate that our method effectively improves the performance of the state-of-the-art model on different datasets.