mlsanityData to 7 Oct 2026

CARETS: A Consistency And Robustness Evaluative Test Suite for VQA

arXiv 2203.07613 ↗First posted 15 Mar 2022cs.CL cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce CARETS, a systematic test suite to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. In contrast to existing VQA test sets, CARETS features balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, especially with concepts such as negation, disjunction, or hypernym invariance. Interestingly, even the most sophisticated models are sensitive to aspects such as swapping the order of terms in a conjunction or varying the number of answer choices mentioned in the question. We release CARETS to be used as an extensible tool for evaluating multi-modal model robustness.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

31 of 46 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
19 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  5. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  6. 2018Bilinear Attention Networkscited by 277
  7. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  8. 2021Generating Datasets with Pretrained Language Modelscited by 192
  9. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  10. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  11. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  12. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  13. 2021A Study of Face Obfuscation in ImageNetcited by 83
  14. 2020Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Textcited by 75
  15. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  16. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  17. 2018Explicit Bias Discovery in Visual Question Answering Modelscited by 42
  18. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  19. 2022Measuring CLEVRness: Blackbox testing of Visual Reasoning Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 2; 2024: 3; 2025: 0; 2026: 1.

11 papers.

  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  3. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  4. 2022Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalizationcited by 20
  5. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  6. 2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
  7. 2022Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systemscited by 6
  8. 2022Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsensecited by 5
  9. 2024BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretrainingcited by 5
  10. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainabilitycited by 1
  11. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0