# Sequence-Level Mixed Sample Data Augmentation

arXiv: https://arxiv.org/abs/2011.09039 · HTML: https://mlsanity.com/paper/2011.09039/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 30 of 71)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2108.06332/) (2021)
- [Learning Algebraic Recombination for Compositional Generalization](https://mlsanity.com/paper/2107.06516/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion](https://mlsanity.com/paper/2503.16212/) (2025)
- [Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization](https://mlsanity.com/paper/2109.02575/) (2021)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)
- [Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models](https://mlsanity.com/paper/2403.02178/) (2024)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2205.06153/) (2022)
- [Self-supervised and Supervised Joint Training for Resource-rich Machine Translation](https://mlsanity.com/paper/2106.04060/) (2021)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [Generating Data for Symbolic Language with Large Language Models](https://mlsanity.com/paper/2305.13917/) (2023)
- [Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks](https://mlsanity.com/paper/2402.13482/) (2024)
- [Grammar-Based Grounded Lexicon Learning](https://mlsanity.com/paper/2202.08806/) (2022)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Delving into the Reversal Curse: How Far Can Large Language Models Generalize?](https://mlsanity.com/paper/2410.18808/) (2024)
- [Mixup Decoding for Diverse Machine Translation](https://mlsanity.com/paper/2109.03402/) (2021)
- [DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification](https://mlsanity.com/paper/2209.05297/) (2022)
- [GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks](https://mlsanity.com/paper/2305.16663/) (2023)

## Abstract (arXiv)

Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language. This work proposes a simple data augmentation approach to encourage compositional behavior in neural models for sequence-to-sequence problems. Our approach, SeqMix, creates new synthetic examples by softly combining input/output sequences from the training set. We connect this approach to existing techniques such as SwitchOut and word dropout, and show that these techniques are all approximating variants of a single objective. SeqMix consistently yields approximately 1.0 BLEU improvement on five different translation datasets over strong Transformer baselines. On tasks that require strong compositional generalization such as SCAN and semantic parsing, SeqMix also offers further improvements.