# CO2Sum:Contrastive Learning for Factual-Consistent Abstractive Summarization

arXiv: https://arxiv.org/abs/2112.01147 · HTML: https://mlsanity.com/paper/2112.01147/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection](https://mlsanity.com/paper/2104.09061/) (2021)
- [Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward](https://mlsanity.com/paper/2005.01159/) (2020)
- [Multi-Fact Correction in Abstractive Text Summarization](https://mlsanity.com/paper/2010.02443/) (2020)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle](https://mlsanity.com/paper/1909.07405/) (2019)

## Built on by (most cited first, 9 of 9)

- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Improving Factuality of Abstractive Summarization via Contrastive Reward Learning](https://mlsanity.com/paper/2307.04507/) (2023)
- [WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs](https://mlsanity.com/paper/2209.13101/) (2022)
- [Factual Dialogue Summarization via Learning from Large Language Models](https://mlsanity.com/paper/2406.14709/) (2024)
- [RAG for Geoscience: What We Expect, Gaps and Opportunities](https://mlsanity.com/paper/2508.11246/) (2025)
- [Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation](https://mlsanity.com/paper/2210.12367/) (2022)
- [Improving abstractive summarization with energy-based re-ranking](https://mlsanity.com/paper/2210.15553/) (2022)
- [Counterfactual Debiasing for Generating Factually Consistent Text Summaries](https://mlsanity.com/paper/2305.10736/) (2023)
- [Context Matters: Pushing the Boundaries of Open-Ended Answer Generation with Graph-Structured Knowledge Context](https://mlsanity.com/paper/2401.12671/) (2024)

## Abstract (arXiv)

Generating factual-consistent summaries is a challenging task for abstractive summarization. Previous works mainly encode factual information or perform post-correct/rank after decoding. In this paper, we provide a factual-consistent solution from the perspective of contrastive learning, which is a natural extension of previous works. We propose CO2Sum (Contrastive for Consistency), a contrastive learning scheme that can be easily applied on sequence-to-sequence models for factual-consistent abstractive summarization, proving that the model can be fact-aware without modifying the architecture. CO2Sum applies contrastive learning on the encoder, which can help the model be aware of the factual information contained in the input article, or performs contrastive learning on the decoder, which makes the model to generate factual-correct output summary. What's more, these two schemes are orthogonal and can be combined to further improve faithfulness. Comprehensive experiments on public benchmarks demonstrate that CO2Sum improves the faithfulness on large pre-trained language models and reaches competitive results compared to other strong factual-consistent summarization baselines.