# Contrastive Learning with Adversarial Perturbations for Conditional Text Generation

arXiv: https://arxiv.org/abs/2012.07280 · HTML: https://mlsanity.com/paper/2012.07280/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 30 of 56)

- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization](https://mlsanity.com/paper/2205.07830/) (2022)
- [Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation](https://mlsanity.com/paper/2109.12242/) (2021)
- [Personalized Showcases: Generating Multi-Modal Explanations for Recommendations](https://mlsanity.com/paper/2207.00422/) (2022)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [CoNT: Contrastive Neural Text Generation](https://mlsanity.com/paper/2205.14690/) (2022)
- [ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues](https://mlsanity.com/paper/2109.00430/) (2021)
- [A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration](https://mlsanity.com/paper/2205.02517/) (2022)
- [ContraCLM: Contrastive Learning For Causal Language Model](https://mlsanity.com/paper/2210.01185/) (2022)
- [CREATER: CTR-driven Advertising Text Generation with Controlled Pre-Training and Contrastive Fine-Tuning](https://mlsanity.com/paper/2205.08943/) (2022)
- [Multi-level Contrastive Learning for Script-based Character Understanding](https://mlsanity.com/paper/2310.13231/) (2023)
- [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022)
- [Neural Machine Translation with Contrastive Translation Memories](https://mlsanity.com/paper/2212.03140/) (2022)
- [Frequency-Aware Contrastive Learning for Neural Machine Translation](https://mlsanity.com/paper/2112.14484/) (2021)
- [Language Detoxification with Attribute-Discriminative Latent Space](https://mlsanity.com/paper/2210.10329/) (2022)
- [Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Information](https://mlsanity.com/paper/2109.05712/) (2021)
- [CO2Sum:Contrastive Learning for Factual-Consistent Abstractive Summarization](https://mlsanity.com/paper/2112.01147/) (2021)
- [Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separately](https://mlsanity.com/paper/2301.11682/) (2023)
- [On the Difficulty of Defending Contrastive Learning against Backdoor Attacks](https://mlsanity.com/paper/2312.09057/) (2023)
- [Reducing Distraction in Long-Context Language Models by Focused Learning](https://mlsanity.com/paper/2411.05928/) (2024)
- [Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning](https://mlsanity.com/paper/2204.04813/) (2022)
- [R2D2: Robust Data-to-Text with Replacement Detection](https://mlsanity.com/paper/2205.12467/) (2022)
- [COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization](https://mlsanity.com/paper/2209.14569/) (2022)
- [Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses](https://mlsanity.com/paper/2305.19339/) (2023)
- [Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation](https://mlsanity.com/paper/2205.13346/) (2022)
- [HiCo: Hierarchical Contrastive Learning for Ultrasound Video Model Pretraining](https://mlsanity.com/paper/2210.04477/) (2022)
- [WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs](https://mlsanity.com/paper/2209.13101/) (2022)
- [EZInterviewer: To Improve Job Interview Performance with Mock Interview Generator](https://mlsanity.com/paper/2301.00972/) (2023)
- [A Topic-aware Summarization Framework with Different Modal Side Information](https://mlsanity.com/paper/2305.11503/) (2023)

## Abstract (arXiv)

Recently, sequence-to-sequence (seq2seq) models with the Transformer architecture have achieved remarkable performance on various conditional text generation tasks, such as machine translation. However, most of them are trained with teacher forcing with the ground truth label given at each time step, without being exposed to incorrectly generated tokens during training, which hurts its generalization to unseen inputs, that is known as the "exposure bias" problem. In this work, we propose to mitigate the conditional text generation problem by contrasting positive pairs with negative pairs, such that the model is exposed to various valid or incorrect perturbations of the inputs, for improved generalization. However, training the model with naive contrastive learning framework using random non-target sequences as negative examples is suboptimal, since they are easily distinguishable from the correct output, especially so with models pretrained with large text corpora. Also, generating positive examples requires domain-specific augmentation heuristics which may not generalize over diverse domains. To tackle this problem, we propose a principled method to generate positive and negative samples for contrastive learning of seq2seq models. Specifically, we generate negative examples by adding small perturbations to the input sequence to minimize its conditional likelihood, and positive examples by adding large perturbations while enforcing it to have a high conditional likelihood. Such "hard" positive and negative pairs generated using our method guides the model to better distinguish correct outputs from incorrect ones. We empirically show that our proposed method significantly improves the generalization of the seq2seq on three text generation tasks - machine translation, text summarization, and question generation.