# Substructure Substitution: Structured Data Augmentation for NLP

arXiv: https://arxiv.org/abs/2101.00411 · HTML: https://mlsanity.com/paper/2101.00411/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 80 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 30 of 48)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://mlsanity.com/paper/2004.11999/) (2020)
- [Logic-Guided Data Augmentation and Regularization for Consistent Question Answering](https://mlsanity.com/paper/2004.10157/) (2020)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://mlsanity.com/paper/2010.11683/) (2020)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Sequence-Level Mixed Sample Data Augmentation](https://mlsanity.com/paper/2011.09039/) (2020)
- [Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding](https://mlsanity.com/paper/1807.01554/) (2018)

## Built on by (most cited first, 25 of 25)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Contrastive Learning Reduces Hallucination in Conversations](https://mlsanity.com/paper/2212.10400/) (2022)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications](https://mlsanity.com/paper/2306.11892/) (2023)
- [SUBS: Subtree Substitution for Compositional Semantic Parsing](https://mlsanity.com/paper/2205.01538/) (2022)
- [TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2205.06153/) (2022)
- [EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classification](https://mlsanity.com/paper/2204.11205/) (2022)
- [Hierarchical Phrase-based Sequence-to-Sequence Learning](https://mlsanity.com/paper/2211.07906/) (2022)
- [Limited Linguistic Diversity in Embodied AI Datasets](https://mlsanity.com/paper/2601.03136/) (2026)
- [Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2110.08538/) (2021)
- [Distribution augmentation for low-resource expressive text-to-speech](https://mlsanity.com/paper/2202.06409/) (2022)
- [WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts](https://mlsanity.com/paper/2401.17703/) (2024)
- [A Unified Framework for Generative Data Augmentation: A Comprehensive Survey](https://mlsanity.com/paper/2310.00277/) (2023)
- [How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models](https://mlsanity.com/paper/2408.16756/) (2024)
- [Data Augmentation for Neural NLP](https://mlsanity.com/paper/2302.11412/) (2023)
- [TreeSwap: Data Augmentation for Machine Translation via Dependency Subtree Swapping](https://mlsanity.com/paper/2311.02355/) (2023)
- [EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries](https://mlsanity.com/paper/2402.13372/) (2024)
- [Learning Language Structures through Grounding](https://mlsanity.com/paper/2406.09662/) (2024)
- [Data Augmentation for Machine Translation via Dependency Subtree Swapping](https://mlsanity.com/paper/2307.07025/) (2023)
- [Feature Weaken: Vicinal Data Augmentation for Classification](https://mlsanity.com/paper/2211.10944/) (2022)
- [SegMix: A Simple Structure-Aware Data Augmentation Method](https://mlsanity.com/paper/2311.09505/) (2023)

## Abstract (arXiv)

We study a family of data augmentation methods, substructure substitution (SUB2), for natural language processing (NLP) tasks. SUB2 generates new examples by substituting substructures (e.g., subtrees or subsequences) with ones with the same label, which can be applied to many structured NLP tasks such as part-of-speech tagging and parsing. For more general tasks (e.g., text classification) which do not have explicitly annotated substructures, we present variations of SUB2 based on constituency parse trees, introducing structure-aware data augmentation methods to general NLP tasks. For most cases, training with the augmented dataset by SUB2 achieves better performance than training with the original training set. Further experiments show that SUB2 has more consistent performance than other investigated augmentation methods, across different tasks and sizes of the seed dataset.