mlsanityData to 7 Oct 2026

Visual Robustness Benchmark for Visual Question Answering (VQA)

arXiv 2407.03386 ↗First posted 3 Jul 2024cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Can Visual Question Answering (VQA) systems perform just as well when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur, which can be detrimental in sensitive applications, such as medical VQA? While linguistic or textual robustness has been thoroughly explored in the VQA literature, there has yet to be any significant work on the visual robustness of VQA models. We propose the first large-scale benchmark comprising 213,000 augmented images, challenging the visual robustness of multiple VQA models and assessing the strength of realistic visual corruptions. Additionally, we have designed several robustness evaluation metrics that can be aggregated into a unified metric and tailored to fit a variety of use cases. Our experiments reveal several insights into the relationships between model size, performance, and robustness with the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising the robustness.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 52 of its 79 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

52 of 79 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2016Universal adversarial perturbationscited by 774
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  10. 2018QuAC : Question Answering in Contextcited by 379
  11. 2016Visual Dialogcited by 377
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  13. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  14. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  15. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  16. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  17. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  18. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  19. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  20. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  21. 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
  22. 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
  23. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  24. 2017Universal Adversarial Perturbations Against Semantic Image Segmentationcited by 92
  25. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  26. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  27. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  28. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  29. 2019Accurate, reliable and fast robustness evaluationcited by 43
  30. 2021A Comprehensive Evaluation Framework for Deep Model Robustnesscited by 41
  31. 2022GRIT: General Robust Image Task Benchmarkcited by 26
  32. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  33. 2018Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captionscited by 22
  34. 2016Zero-Shot Visual Question Answeringcited by 20
  35. 2017A Novel Framework for Robustness Analysis of Visual QA Modelscited by 20
  36. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15
  37. 2021Visual Question Answering: which investigated applications?cited by 11
  38. 2022CARETS: A Consistency And Robustness Evaluative Test Suite for VQAcited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 6.

12 papers.

  1. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  2. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 10
  3. 2025SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoningcited by 7
  4. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3
  5. 2026Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specificationcited by 2
  6. 2026Same Answer, Different Representations: Hidden instability in VLMscited by 2
  7. 2026RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenescited by 2
  8. 2025VModA: An Effective Framework for Adaptive NSFW Image Moderationcited by 1
  9. 2024Contextual Breach: Assessing the Robustness of Transformer-based QA Modelscited by 0
  10. 2026FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failurescited by 0
  11. 2026Latent Denoising Improves Visual Alignment in Large Multimodal Modelscited by 0
  12. 2026RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistantscited by 0