mlsanityData to 7 Oct 2026

Human-Adversarial Visual Question Answering

arXiv 2106.02280 ↗First posted 4 Jun 2021cs.CV cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, it is clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against human-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model's predicted answer is incorrect. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 54 of its 64 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

54 of 64 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  13. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  14. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  15. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  16. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  17. 2019Scene Text Visual Question Answeringcited by 327
  18. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  19. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  20. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  21. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  22. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  23. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  24. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  25. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  26. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  27. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  28. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  29. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  30. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  31. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  32. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  33. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  34. 2019Rosetta: Large scale system for text detection and recognition in imagescited by 69
  35. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  36. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  37. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 61
  38. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 13; 2024: 12; 2025: 10; 2026: 6.

57 papers.

Sort:

By area:

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  2. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  3. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  4. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  5. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  6. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  7. 2022Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformercited by 58
  8. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  9. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Modelscited by 40
  10. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Modelscited by 27
  11. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24
  12. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 24
38 more
  1. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  2. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  3. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  4. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  5. 2022Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalizationcited by 20
  6. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  7. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  8. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  9. 2022Dynatask: A Framework for Creating Dynamic AI Benchmark Taskscited by 16
  10. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  11. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  12. 2024Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrievalcited by 11
  13. 2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 11
  14. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 10
  15. 2021Semantically Distributed Robust Optimization for Vision-and-Language Inferencecited by 9
  16. 2022MaXM: Towards Multilingual Visual Question Answeringcited by 9
  17. 2023Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questionscited by 9
  18. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answeringcited by 7
  19. 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Modelscited by 7
  20. 2024Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialnesscited by 7
  21. 2021Unsupervised Natural Language Inference Using PHL Triplet Generationcited by 6
  22. 2021MLP Architectures for Vision-and-Language Modeling: An Empirical Studycited by 6
  23. 2023EVJVQA Challenge: Multilingual Visual Question Answeringcited by 6
  24. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMscited by 6
  25. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Predictioncited by 5
  26. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  27. 2022Fine-Grained Visual Entailmentcited by 4
  28. 2022Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutionscited by 4
  29. 2025Mitigating Easy Option Bias in Multiple-Choice Question Answeringcited by 4
  30. 2022Measuring CLEVRness: Blackbox testing of Visual Reasoning Modelscited by 3
  31. 2023Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learningcited by 3
  32. 2026Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probescited by 2
  33. 2021Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity Resolutioncited by 1
  34. 2024How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generationcited by 1
  35. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  36. 2025Differences That Matter: Auditing Models for Capability Gap Discovery and Rectificationcited by 1
  37. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  38. 2026AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QAcited by 1

50 most cited of 57 All 57 (JSON)