# On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study

arXiv: https://arxiv.org/abs/2106.00872 · HTML: https://mlsanity.com/paper/2106.00872/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://mlsanity.com/paper/1810.12885/) (2018)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension](https://mlsanity.com/paper/1804.07927/) (2018)

## Built on by (most cited first, 30 of 31)

- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Supporting Human-AI Collaboration in Auditing LLMs with LLMs](https://mlsanity.com/paper/2304.09991/) (2023)
- [An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models](https://mlsanity.com/paper/2403.09766/) (2024)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [Retrieval-guided Counterfactual Generation for QA](https://mlsanity.com/paper/2110.07596/) (2021)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)
- [CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation](https://mlsanity.com/paper/2210.04873/) (2022)
- [MirrorCheck: Efficient Adversarial Defense for Vision-Language Models](https://mlsanity.com/paper/2406.09250/) (2024)
- [Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks](https://mlsanity.com/paper/2204.01906/) (2022)
- [A Theory of Dynamic Benchmarks](https://mlsanity.com/paper/2210.03165/) (2022)
- [Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair](https://mlsanity.com/paper/2111.08181/) (2021)
- [Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition](https://mlsanity.com/paper/2107.13935/) (2021)
- [What Makes Reading Comprehension Questions Difficult?](https://mlsanity.com/paper/2203.06342/) (2022)
- [AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization](https://mlsanity.com/paper/2212.00921/) (2022)
- [Benchmarking Long-tail Generalization with Likelihood Splits](https://mlsanity.com/paper/2210.06799/) (2022)
- [longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks](https://mlsanity.com/paper/2206.14729/) (2022)
- [Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness](https://mlsanity.com/paper/2406.16342/) (2024)
- [Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models](https://mlsanity.com/paper/2305.06841/) (2023)
- [Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases](https://mlsanity.com/paper/2305.18917/) (2023)
- [Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios](https://mlsanity.com/paper/2209.07760/) (2022)
- [Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity Resolution](https://mlsanity.com/paper/2111.10497/) (2021)
- [Resolving the Human Subjects Status of Machine Learning's Crowdworkers](https://mlsanity.com/paper/2206.04039/) (2022)
- [Bridging the Training-Inference Gap for Dense Phrase Retrieval](https://mlsanity.com/paper/2210.13678/) (2022)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop](https://mlsanity.com/paper/2206.14272/) (2022)
- [Implicit Design Choices and Their Impact on Emotion Recognition Model Development and Evaluation](https://mlsanity.com/paper/2309.03238/) (2023)
- [Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models](https://mlsanity.com/paper/2506.22982/) (2025)
- [Enhancing the QA Model through a Multi-domain Debiasing Framework](https://mlsanity.com/paper/2601.11581/) (2026)

## Abstract (arXiv)

In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that models trained on these more challenging datasets will rely less on superficial patterns, and thus be less brittle. However, despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models. In this paper, we conduct a large-scale controlled study focused on question answering, assigning workers at random to compose questions either (i) adversarially (with a model in the loop); or (ii) in the standard fashion (without a model). Across a variety of models and datasets, we find that models trained on adversarial data usually perform better on other adversarial datasets but worse on a diverse collection of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of adversarial (vs standard) data, identifying key differences and offering guidance for future research.