# Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

arXiv: https://arxiv.org/abs/2104.08678 · HTML: https://mlsanity.com/paper/2104.08678/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [DynaSent: A Dynamic Benchmark for Sentiment Analysis](https://mlsanity.com/paper/2012.15349/) (2020)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)

## Built on by (most cited first, 30 of 79)

- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Explore, Establish, Exploit: Red Teaming Language Models from Scratch](https://mlsanity.com/paper/2306.09442/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [Query-Efficient Black-Box Red Teaming via Bayesian Optimization](https://mlsanity.com/paper/2305.17444/) (2023)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [Retrieval-guided Counterfactual Generation for QA](https://mlsanity.com/paper/2110.07596/) (2021)
- [Protecting Your LLMs with Information Bottleneck](https://mlsanity.com/paper/2404.13968/) (2024)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains](https://mlsanity.com/paper/2410.17952/) (2024)
- [QAmeleon: Multilingual QA with Only 5 Examples](https://mlsanity.com/paper/2211.08264/) (2022)
- [CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation](https://mlsanity.com/paper/2210.04873/) (2022)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [Question Answering Infused Pre-training of General-Purpose Contextualized Representations](https://mlsanity.com/paper/2106.08190/) (2021)
- [MirrorCheck: Efficient Adversarial Defense for Vision-Language Models](https://mlsanity.com/paper/2406.09250/) (2024)
- [Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization](https://mlsanity.com/paper/2405.09113/) (2024)
- [Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games](https://mlsanity.com/paper/2310.00322/) (2023)
- [Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks](https://mlsanity.com/paper/2204.01906/) (2022)
- [An Empirical Comparison of LM-based Question and Answer Generation Methods](https://mlsanity.com/paper/2305.17002/) (2023)
- [Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions](https://mlsanity.com/paper/2504.11373/) (2025)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset](https://mlsanity.com/paper/2403.19559/) (2024)

## Abstract (arXiv)

Despite recent progress, state-of-the-art question answering models remain vulnerable to a variety of adversarial attacks. While dynamic adversarial data collection, in which a human annotator tries to write examples that fool a model-in-the-loop, can improve model robustness, this process is expensive which limits the scale of the collected data. In this work, we are the first to use synthetic adversarial data generation to make question answering models more robust to human adversaries. We develop a data generation pipeline that selects source passages, identifies candidate answers, generates questions, then finally filters or re-labels them to improve quality. Using this approach, we amplify a smaller human-written adversarial dataset to a much larger set of synthetic question-answer pairs. By incorporating our synthetic data, we improve the state-of-the-art on the AdversarialQA dataset by 3.7F1 and improve model generalisation on nine of the twelve MRQA datasets. We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8.8% of the time on average, compared to 17.6% for a model trained without synthetic data.