mlsanityData to 7 Oct 2026

CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations

arXiv 2204.02380 ↗First posted 5 Apr 2022cs.CV cs.CLcited by 16

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generating natural language explanations for VQA, we introduce the large-scale CLEVR-X dataset that extends the CLEVR dataset with natural language explanations. For each image-question pair in the CLEVR dataset, CLEVR-X contains multiple structured textual explanations which are derived from the original scene graphs. By construction, the CLEVR-X explanations are correct and describe the reasoning and visual information that is necessary to answer a given question. We conducted a user study to confirm that the ground-truth explanations in our proposed dataset are indeed complete and relevant. We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset. Furthermore, we provide a detailed analysis of the explanation generation quality for different question and answer types. Additionally, we study the influence of using different numbers of ground-truth explanations on the convergence of natural language generation (NLG) metrics. The CLEVR-X dataset is publicly available at urlhttps://explainableml.github.io/CLEVR-X/.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 61 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

52 of 61 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
38 more
  1. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  2. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  3. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  6. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  7. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  8. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  9. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  10. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  11. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  12. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  13. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  14. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  15. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  16. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  17. 2020Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAIcited by 125
  18. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  19. 2019Learning by Abstraction: The Neural State Machinecited by 113
  20. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  21. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  22. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  23. 2018Grounding Visual Explanationscited by 92
  24. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  25. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  26. 2015Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answeringcited by 66
  27. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64
  28. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  29. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  30. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  31. 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
  32. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  33. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  34. 2020CURI: A Benchmark for Productive Concept Learning Under Uncertaintycited by 21
  35. 2017Interpretable Counting for Visual Question Answeringcited by 17
  36. 2021Keep CALM and Improve Visual Feature Attributioncited by 15
  37. 2020Robust Explanations for Visual Question Answeringcited by 14
  38. 2020Improving VQA and its Explanations by Comparing Competing Explanationscited by 12

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 3; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  3. 2023No Representation Rules Them All in Category Discoverycited by 61
  4. 2023Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulationcited by 60
  5. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  6. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  7. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 20
  8. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  9. 2024The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systemscited by 6
  10. 2022ILLUME: Rationalizing Vision-Language Models through Human Interactionscited by 5
  11. 2023ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamesecited by 5
  12. 2024Study on the Helpfulness of Explainable Artificial Intelligencecited by 4
4 more
  1. 2023Zero-shot Translation of Attention Patterns in VQA Models to Natural Languagecited by 3
  2. 2022Towards Reasoning-Aware Explainable VQAcited by 2
  3. 2025Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Modelscited by 1
  4. 2026When Do Diffusion Models learn to Generate Multiple Objects?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images
  2. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  4. 2023Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks
  5. 2021'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks
  6. 2023Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering
  7. 2022QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning
  8. 2025NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks