# CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

arXiv: https://arxiv.org/abs/2010.08670 · HTML: https://mlsanity.com/paper/2010.08670/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 30 of 41)

- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives](https://mlsanity.com/paper/2206.02457/) (2022)
- [Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning](https://mlsanity.com/paper/2109.05941/) (2021)
- [Task Compass: Scaling Multi-task Pre-training with Task Prefix](https://mlsanity.com/paper/2210.06277/) (2022)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [Fact Checking with Insufficient Evidence](https://mlsanity.com/paper/2204.02007/) (2022)
- [Improving Short Text Classification With Augmented Data Using GPT-3](https://mlsanity.com/paper/2205.10981/) (2022)
- [Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax](https://mlsanity.com/paper/2105.13608/) (2021)
- [Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses](https://mlsanity.com/paper/2305.19339/) (2023)
- [Contrastive Document Representation Learning with Graph Attention Networks](https://mlsanity.com/paper/2110.10778/) (2021)
- [Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning](https://mlsanity.com/paper/2312.14222/) (2023)
- [Exploring Empty Spaces: Human-in-the-Loop Data Augmentation](https://mlsanity.com/paper/2410.01088/) (2024)
- [Unsupervised Document Embedding via Contrastive Augmentation](https://mlsanity.com/paper/2103.14542/) (2021)
- [When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation](https://mlsanity.com/paper/2203.09391/) (2022)
- [COMICS: End-to-end Bi-grained Contrastive Learning for Multi-face Forgery Detection](https://mlsanity.com/paper/2308.01520/) (2023)
- [CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation](https://mlsanity.com/paper/2204.07674/) (2022)
- [Domain Confused Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2207.04564/) (2022)
- [PhysioSync: Temporal and Cross-Modal Contrastive Learning Inspired by Physiological Synchronization for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2504.17163/) (2025)
- [A Benchmark Study of Contrastive Learning for Arabic Social Meaning](https://mlsanity.com/paper/2210.12314/) (2022)
- [Semantic-aware Contrastive Learning for More Accurate Semantic Parsing](https://mlsanity.com/paper/2301.07919/) (2023)
- ["Nothing Abnormal": Disambiguating Medical Reports via Contrastive Knowledge Infusion](https://mlsanity.com/paper/2305.08300/) (2023)
- [MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teaching](https://mlsanity.com/paper/2506.02689/) (2025)
- [Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation](https://mlsanity.com/paper/2510.09599/) (2025)
- [Attention-based Contrastive Learning for Winograd Schemas](https://mlsanity.com/paper/2109.05108/) (2021)

## Abstract (arXiv)

Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation framework dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization objective is introduced to capture the global relationship among all the data samples. A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2% while applied to the RoBERTa-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training base-lines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.