# Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI

arXiv: https://arxiv.org/abs/2003.07258 · HTML: https://mlsanity.com/paper/2003.07258/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2020.

## Built on by (most cited first, 30 of 126)

- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare](https://mlsanity.com/paper/2309.12325/) (2023)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023)
- [Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining](https://mlsanity.com/paper/2110.08412/) (2021)
- [SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability](https://mlsanity.com/paper/2208.09418/) (2022)
- [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science](https://mlsanity.com/paper/2303.00652/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations](https://mlsanity.com/paper/2211.12486/) (2022)
- [Position: Explain to Question not to Justify](https://mlsanity.com/paper/2402.13914/) (2024)
- [An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability](https://mlsanity.com/paper/2109.05327/) (2021)
- [CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations](https://mlsanity.com/paper/2204.02380/) (2022)
- [Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics](https://mlsanity.com/paper/2409.16756/) (2024)
- [Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasets](https://mlsanity.com/paper/2311.01961/) (2023)
- [Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset](https://mlsanity.com/paper/2204.11642/) (2022)
- [RELAX: Representation Learning Explainability](https://mlsanity.com/paper/2112.10161/) (2021)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Optimizing Explanations by Network Canonization and Hyperparameter Search](https://mlsanity.com/paper/2211.17174/) (2022)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology](https://mlsanity.com/paper/2204.05044/) (2022)
- [Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learning](https://mlsanity.com/paper/2205.03398/) (2022)
- [Uncovering the Full Potential of Visual Grounding Methods in VQA](https://mlsanity.com/paper/2401.07803/) (2024)
- [Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification](https://mlsanity.com/paper/2306.12150/) (2023)

## Abstract (arXiv)

The rise of deep learning in today's applications entailed an increasing need in explaining the model's decisions beyond prediction performances in order to foster trust and accountability. Recently, the field of explainable AI (XAI) has developed methods that provide such explanations for already trained neural networks. In computer vision tasks such explanations, termed heatmaps, visualize the contributions of individual pixels to the prediction. So far XAI methods along with their heatmaps were mainly validated qualitatively via human-based assessment, or evaluated through auxiliary proxy tasks such as pixel perturbation, weak object localization or randomization tests. Due to the lack of an objective and commonly accepted quality measure for heatmaps, it was debatable which XAI method performs best and whether explanations can be trusted at all. In the present work, we tackle the problem by proposing a ground truth based evaluation framework for XAI methods based on the CLEVR visual question answering task. Our framework provides a (1) selective, (2) controlled and (3) realistic testbed for the evaluation of neural network explanations. We compare ten different explanation methods, resulting in new insights about the quality and properties of XAI methods, sometimes contradicting with conclusions from previous comparative studies. The CLEVR-XAI dataset and the benchmarking code can be found at https://github.com/ahmedmagdiosman/clevr-xai.