# Learning to Perturb Word Embeddings for Out-of-distribution QA

arXiv: https://arxiv.org/abs/2105.02692 · HTML: https://mlsanity.com/paper/2105.02692/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation](https://mlsanity.com/paper/2001.08735/) (2020)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [The Effect of Natural Distribution Shift on Question Answering Models](https://mlsanity.com/paper/2004.14444/) (2020)
- [SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness](https://mlsanity.com/paper/2009.10195/) (2020)
- [Edinburgh Neural Machine Translation Systems for WMT 16](https://mlsanity.com/paper/1606.02891/) (2016)
- [Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering](https://mlsanity.com/paper/1909.06356/) (2019)
- [Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia](https://mlsanity.com/paper/1805.05942/) (2018)
- [Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs](https://mlsanity.com/paper/2005.13837/) (2020)
- [Domain-agnostic Question-Answering with Adversarial Training](https://mlsanity.com/paper/1910.09342/) (2019)
- [Meta Back-translation](https://mlsanity.com/paper/2102.07847/) (2021)

## Built on by (most cited first, 13 of 13)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [KALA: Knowledge-Augmented Language Model Adaptation](https://mlsanity.com/paper/2204.10555/) (2022)
- [HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models](https://mlsanity.com/paper/2410.01524/) (2024)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation](https://mlsanity.com/paper/2203.07735/) (2022)
- [Unsupervised Question Answering via Answer Diversifying](https://mlsanity.com/paper/2208.10813/) (2022)
- [A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling](https://mlsanity.com/paper/2306.14806/) (2023)
- [Improving Cooperation in Language Games with Bayesian Inference and the Cognitive Hierarchy](https://mlsanity.com/paper/2412.12409/) (2024)
- [Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling](https://mlsanity.com/paper/2302.08659/) (2023)
- [Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models](https://mlsanity.com/paper/2411.00686/) (2024)
- [Using calibrator to improve robustness in Machine Reading Comprehension](https://mlsanity.com/paper/2202.11865/) (2022)
- [FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings](https://mlsanity.com/paper/2505.21848/) (2025)

## Abstract (arXiv)

QA models based on pretrained language mod-els have achieved remarkable performance on various benchmark datasets.However, QA models do not generalize well to unseen data that falls outside the training distribution, due to distributional shifts.Data augmentation (DA) techniques which drop/replace words have shown to be effective in regularizing the model from overfitting to the training data.Yet, they may adversely affect the QA tasks since they incur semantic changes that may lead to wrong answers for the QA task. To tackle this problem, we propose a simple yet effective DA method based on a stochastic noise generator, which learns to perturb the word embedding of the input questions and context without changing their semantics. We validate the performance of the QA models trained with our word embedding perturbation on a single source dataset, on five different target domains.The results show that our method significantly outperforms the baselineDA methods. Notably, the model trained with ours outperforms the model trained with more than 240K artificially generated QA pairs.