# SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization

arXiv: https://arxiv.org/abs/2106.01890 · HTML: https://mlsanity.com/paper/2106.01890/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [RefSum: Refactoring Neural Summarization](https://mlsanity.com/paper/2104.07210/) (2021)
- [Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization](https://mlsanity.com/paper/1909.00141/) (2019)

## Built on by (most cited first, 30 of 132)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization](https://mlsanity.com/paper/2205.07830/) (2022)
- [Detecting and Mitigating Hallucinations in Multilingual Summarisation](https://mlsanity.com/paper/2305.13632/) (2023)
- [Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing](https://mlsanity.com/paper/2405.00467/) (2024)
- [Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learning](https://mlsanity.com/paper/2204.08143/) (2022)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Routoo: Learning to Route to Large Language Models Effectively](https://mlsanity.com/paper/2401.13979/) (2024)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [Sequence Level Contrastive Learning for Text Summarization](https://mlsanity.com/paper/2109.03481/) (2021)
- [Faithfulness-Aware Decoding Strategies for Abstractive Summarization](https://mlsanity.com/paper/2303.03278/) (2023)
- [Towards Improving Faithfulness in Abstractive Summarization](https://mlsanity.com/paper/2210.01877/) (2022)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)

## Abstract (arXiv)

In this paper, we present a conceptually simple while empirically powerful framework for abstractive summarization, SimCLS, which can bridge the gap between the learning objective and evaluation metrics resulting from the currently dominated sequence-to-sequence learning framework by formulating text generation as a reference-free evaluation problem (i.e., quality estimation) assisted by contrastive learning. Experimental results show that, with minor modification over existing top-scoring systems, SimCLS can improve the performance of existing top-performing models by a large margin. Particularly, 2.51 absolute improvement against BART and 2.50 over PEGASUS w.r.t ROUGE-1 on the CNN/DailyMail dataset, driving the state-of-the-art performance to a new level. We have open-sourced our codes and results: https://github.com/yixinL7/SimCLS. Results of our proposed models have been deployed into ExplainaBoard platform, which allows researchers to understand our systems in a more fine-grained way.