# Contrastive Attention Mechanism for Abstractive Sentence Summarization

arXiv: https://arxiv.org/abs/1910.13114 · HTML: https://mlsanity.com/paper/1910.13114/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 13 of 13)

- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding](https://mlsanity.com/paper/2309.08929/) (2023)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Adversarial Training with Contrastive Learning in NLP](https://mlsanity.com/paper/2109.09075/) (2021)
- [Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism](https://mlsanity.com/paper/2109.15153/) (2021)
- [DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization](https://mlsanity.com/paper/2112.08414/) (2021)
- [Improving Radiology Summarization with Radiograph and Anatomy Prompts](https://mlsanity.com/paper/2210.08303/) (2022)
- [Controlled Text Generation with Hidden Representation Transformations](https://mlsanity.com/paper/2305.19230/) (2023)
- [Domain Aligned Prefix Averaging for Domain Generalization in Abstractive Summarization](https://mlsanity.com/paper/2305.16820/) (2023)
- [From Lengthy to Lucid: A Systematic Literature Review on NLP Techniques for Taming Long Sentences](https://mlsanity.com/paper/2312.05172/) (2023)
- [ConVerSum: A Contrastive Learning-based Approach for Data-Scarce Solution of Cross-Lingual Summarization Beyond Direct Equivalents](https://mlsanity.com/paper/2408.09273/) (2024)
- [Comparative Analysis of Abstractive Summarization Models for Clinical Radiology Reports](https://mlsanity.com/paper/2506.16247/) (2025)

## Abstract (arXiv)

We propose a contrastive attention mechanism to extend the sequence-to-sequence framework for abstractive sentence summarization task, which aims to generate a brief summary of a given source sentence. The proposed contrastive attention mechanism accommodates two categories of attention: one is the conventional attention that attends to relevant parts of the source sentence, the other is the opponent attention that attends to irrelevant or less relevant parts of the source sentence. Both attentions are trained in an opposite way so that the contribution from the conventional attention is encouraged and the contribution from the opponent attention is discouraged through a novel softmax and softmin functionality. Experiments on benchmark datasets show that, the proposed contrastive attention mechanism is more focused on the relevant parts for the summary than the conventional attention mechanism, and greatly advances the state-of-the-art performance on the abstractive sentence summarization task. We release the code at https://github.com/travel-go/Abstractive-Text-Summarization