mlsanityData to 14 Oct 2026

longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks

arXiv 2206.14729 ↗First posted 29 Jun 2022cs.CL cs.AI cs.HCcited by 8

Not on the map. Why?

Abstract (arXiv)

Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team "longhorns" on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

26 of 47 references.

References · 26

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  9. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  10. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  11. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  12. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
14 more
  1. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  2. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  3. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 84
  4. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
  5. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  6. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  7. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  8. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  9. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  10. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 31
  11. 2020ConjNLI: Natural Language Inference Over Conjunctive Sentencescited by 27
  12. 2018Analyzing Compositionality-Sensitivity of NLI Modelscited by 26
  13. 2021Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Faircited by 12
  14. 2020"What is on your mind?" Automated Scoring of Mindreading in Childhood and Early Adolescencecited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 1.

8 papers.

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  2. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Modelscited by 45
  3. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  4. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  5. 2023Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Modelscited by 6
  6. 2024Benchmark Transparency: Measuring the Impact of Data on Evaluationcited by 4
  7. 2022InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examplescited by 3
  8. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0