# Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks

arXiv: https://arxiv.org/abs/2010.02394 · HTML: https://mlsanity.com/paper/2010.02394/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 15 of 15)

- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [User Preference-aware Fake News Detection](https://mlsanity.com/paper/2104.12259/) (2021)
- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition](https://mlsanity.com/paper/2205.04442/) (2022)
- [Exploring Representation-Level Augmentation for Code Search](https://mlsanity.com/paper/2210.12285/) (2022)
- [LINDA: Unsupervised Learning to Interpolate in Natural Language Processing](https://mlsanity.com/paper/2112.13969/) (2021)
- [Provable Benefit of Mixup for Finding Optimal Decision Boundaries](https://mlsanity.com/paper/2306.00267/) (2023)
- [A Data Cartography based MixUp for Pre-trained Language Models](https://mlsanity.com/paper/2205.03403/) (2022)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence](https://mlsanity.com/paper/2207.11076/) (2022)
- [Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup](https://mlsanity.com/paper/2109.07177/) (2021)
- [A Data Augmentation Method for Fully Automatic Brain Tumor Segmentation](https://mlsanity.com/paper/2202.06344/) (2022)
- [Constructing Contrastive samples via Summarization for Text Classification with limited annotations](https://mlsanity.com/paper/2104.05094/) (2021)
- [AttentionMix: Data augmentation method that relies on BERT attention mechanism](https://mlsanity.com/paper/2309.11104/) (2023)
- [MSMix:An Interpolation-Based Text Data Augmentation Method Manifold Swap Mixup](https://mlsanity.com/paper/2305.19617/) (2023)

## Abstract (arXiv)

Mixup is the latest data augmentation technique that linearly interpolates input examples and the corresponding labels. It has shown strong effectiveness in image classification by interpolating images at the pixel level. Inspired by this line of research, in this paper, we explore i) how to apply mixup to natural language processing tasks since text data can hardly be mixed in the raw format; ii) if mixup is still effective in transformer-based learning models, e.g., BERT. To achieve the goal, we incorporate mixup to transformer-based pre-trained architecture, named "mixup-transformer", for a wide range of NLP tasks while keeping the whole end-to-end training system. We evaluate the proposed framework by running extensive experiments on the GLUE benchmark. Furthermore, we also examine the performance of mixup-transformer in low-resource scenarios by reducing the training data with a certain ratio. Our studies show that mixup is a domain-independent data augmentation technique to pre-trained language models, resulting in significant performance improvement for transformer-based models.