mlsanityData to 7 Oct 2026

Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

arXiv 2212.00259 ↗First posted 1 Dec 2022cs.CV cs.CLcited by 90

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 70 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

54 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations5 Apr

Other references · 53

  1. 2017Mask R-CNNcited by 7,259
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
38 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  6. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  7. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  10. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  11. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  12. 2020Learning to Learn Single Domain Generalizationcited by 295
  13. 2018Bilinear Attention Networkscited by 277
  14. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  15. 2019Episodic Training for Domain Generalizationcited by 245
  16. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  17. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  18. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  19. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  20. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  21. 2015Exploring Models and Data for Image Question Answeringcited by 139
  22. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  23. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  24. 2016Range Loss for Deep Face Recognition with Long-tailcited by 111
  25. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  26. 2021Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigationcited by 78
  27. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  28. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  29. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  30. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  31. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  32. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  33. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  34. 2021Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognitioncited by 27
  35. 2018Cross-Dataset Adaptation for Visual Question Answeringcited by 17
  36. 2021Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehiclescited by 16
  37. 2021How Transferable are Reasoning Patterns in VQA?cited by 16
  38. 2021Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Imagescited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 26; 2025: 43; 2026: 16.

90 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  5. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  8. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  9. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  10. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  11. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  12. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
38 more
  1. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  2. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  3. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  4. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  5. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  6. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  7. 2023No Representation Rules Them All in Category Discoverycited by 61
  8. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  9. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  10. 2025FineVision: Open Data Is All You Needcited by 51
  11. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  12. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  13. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  14. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
  15. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  16. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  17. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  18. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  19. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  20. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  21. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  22. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 20
  23. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  24. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  25. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 18
  26. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answeringcited by 18
  27. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  28. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  29. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  30. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 16
  31. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 15
  32. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  33. 2025LLaVA-c: Continual Improved Visual Instruction Tuningcited by 14
  34. 2024Learning to Compose: Improving Object Centric Learning by Injecting Compositionalitycited by 13
  35. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  36. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13
  37. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  38. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization
  2. 2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering
  3. 2025Evaluating Variance in Visual Question Answering Benchmarks
  4. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  5. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  6. 2025CATCH: A Modular Cross-domain Adaptive Template with Hook
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  8. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images