# A Novel Framework for Robustness Analysis of Visual QA Models

arXiv: https://arxiv.org/abs/1711.06232 · HTML: https://mlsanity.com/paper/1711.06232/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2017.

## Built on by (most cited first, 20 of 20)

- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics](https://mlsanity.com/paper/2406.15264/) (2024)
- [Contextualized Keyword Representations for Multi-modal Retinal Image Captioning](https://mlsanity.com/paper/2104.12471/) (2021)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Query-based Video Summarization with Pseudo Label Supervision](https://mlsanity.com/paper/2307.01945/) (2023)
- [Visual Robustness Benchmark for Visual Question Answering (VQA)](https://mlsanity.com/paper/2407.03386/) (2024)
- [Domain-robust VQA with diverse datasets and methods but no target labels](https://mlsanity.com/paper/2103.15974/) (2021)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions](https://mlsanity.com/paper/2304.03147/) (2023)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Causalainer: Causal Explainer for Automatic Video Summarization](https://mlsanity.com/paper/2305.00455/) (2023)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models](https://mlsanity.com/paper/2407.19041/) (2024)
- [Causal Video Summarizer for Video Exploration](https://mlsanity.com/paper/2307.01947/) (2023)
- [Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking](https://mlsanity.com/paper/2110.05159/) (2021)
- [A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation](https://mlsanity.com/paper/2408.12398/) (2024)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)

## Abstract (arXiv)

Deep neural networks have been playing an essential role in many computer vision tasks including Visual Question Answering (VQA). Until recently, the study of their accuracy was the main focus of research but now there is a trend toward assessing the robustness of these models against adversarial attacks by evaluating their tolerance to varying noise levels. In VQA, adversarial attacks can target the image and/or the proposed main question and yet there is a lack of proper analysis of the later. In this work, we propose a flexible framework that focuses on the language part of VQA that uses semantically relevant questions, dubbed basic questions, acting as controllable noise to evaluate the robustness of VQA models. We hypothesize that the level of noise is positively correlated to the similarity of a basic question to the main question. Hence, to apply noise on any given main question, we rank a pool of basic questions based on their similarity by casting this ranking task as a LASSO optimization problem. Then, we propose a novel robustness measure, R_score, and two large-scale basic question datasets (BQDs) in order to standardize robustness analysis for VQA models.