# SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

arXiv: https://arxiv.org/abs/2009.10195 · HTML: https://mlsanity.com/paper/2009.10195/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 30 of 96)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Towards Improving Adversarial Training of NLP Models](https://mlsanity.com/paper/2109.00544/) (2021)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [Perturbation Augmentation for Fairer NLP](https://mlsanity.com/paper/2205.12586/) (2022)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)
- [SSMix: Saliency-Based Span Mixup for Text Classification](https://mlsanity.com/paper/2106.08062/) (2021)
- [DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation](https://mlsanity.com/paper/2202.12350/) (2022)
- [FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2108.06332/) (2021)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [Mixture of Soft Prompts for Controllable Data Generation](https://mlsanity.com/paper/2303.01580/) (2023)
- [CoNT: Contrastive Neural Text Generation](https://mlsanity.com/paper/2205.14690/) (2022)
- [GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation](https://mlsanity.com/paper/2109.03079/) (2021)
- [HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models](https://mlsanity.com/paper/2410.01524/) (2024)
- [Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems](https://mlsanity.com/paper/2108.12589/) (2021)
- [TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2205.06153/) (2022)
- [Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation](https://mlsanity.com/paper/2210.02952/) (2022)
- [Learning to Perturb Word Embeddings for Out-of-distribution QA](https://mlsanity.com/paper/2105.02692/) (2021)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models](https://mlsanity.com/paper/2310.09624/) (2023)
- [Estimating Soft Labels for Out-of-Domain Intent Detection](https://mlsanity.com/paper/2211.05561/) (2022)
- [Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax](https://mlsanity.com/paper/2105.13608/) (2021)
- [ALP: Data Augmentation using Lexicalized PCFGs for Few-Shot Text Classification](https://mlsanity.com/paper/2112.11916/) (2021)
- [On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2104.01393/) (2021)
- [Consistency Training with Virtual Adversarial Discrete Perturbation](https://mlsanity.com/paper/2104.07284/) (2021)

## Abstract (arXiv)

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English.