# FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methods

arXiv: https://arxiv.org/abs/2308.06248 · HTML: https://mlsanity.com/paper/2308.06248/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 86 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)

## Built on by (most cited first, 30 of 50)

- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images](https://mlsanity.com/paper/2312.02253/) (2023)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics](https://mlsanity.com/paper/2409.16756/) (2024)
- [ProtoS-ViT: Visual foundation models for sparse self-explainable classifications](https://mlsanity.com/paper/2406.10025/) (2024)
- [Benchmarking the Attribution Quality of Vision Models](https://mlsanity.com/paper/2407.11910/) (2024)
- [InfoDisent: Explainability of Image Classification Models by Information Disentanglement](https://mlsanity.com/paper/2409.10329/) (2024)
- [FungiTastic: A multi-modal dataset and benchmark for image categorization](https://mlsanity.com/paper/2408.13632/) (2024)
- [EPIC: Explanation of Pretrained Image Classification Networks via Prototype](https://mlsanity.com/paper/2505.12897/) (2025)
- [Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations](https://mlsanity.com/paper/2310.08537/) (2023)
- [Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers](https://mlsanity.com/paper/2507.04388/) (2025)
- [SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions](https://mlsanity.com/paper/2507.23784/) (2025)
- [Reactive Model Correction: Mitigating Harm to Task-Relevant Features via Conditional Bias Suppression](https://mlsanity.com/paper/2404.09601/) (2024)
- [On the Evaluation Consistency of Attribution-based Explanations](https://mlsanity.com/paper/2407.19471/) (2024)
- [LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions](https://mlsanity.com/paper/2411.16760/) (2024)
- [Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantification](https://mlsanity.com/paper/2411.07097/) (2024)
- [SIDE: Sparse Information Disentanglement for Explainable Artificial Intelligence](https://mlsanity.com/paper/2507.19321/) (2025)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [This looks like what? Challenges and Future Research Directions for Part-Prototype Models](https://mlsanity.com/paper/2502.09340/) (2025)
- [Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning](https://mlsanity.com/paper/2503.08636/) (2025)
- [Personalized Interpretability -- Interactive Alignment of Prototypical Parts Networks](https://mlsanity.com/paper/2506.05533/) (2025)
- [ComFe: An Interpretable Head for Vision Transformers](https://mlsanity.com/paper/2403.04125/) (2024)
- [Revisiting FunnyBirds evaluation framework for prototypical parts networks](https://mlsanity.com/paper/2408.11401/) (2024)
- [OMENN: One Matrix to Explain Neural Networks](https://mlsanity.com/paper/2412.02399/) (2024)
- [Tell me why: Visual foundation models as self-explainable classifiers](https://mlsanity.com/paper/2502.19577/) (2025)
- [Locally Explaining Prediction Behavior via Gradual Interventions and Measuring Property Gradients](https://mlsanity.com/paper/2503.05424/) (2025)
- [FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI](https://mlsanity.com/paper/2511.15481/) (2025)
- [Data Science Principles for Interpretable and Explainable AI](https://mlsanity.com/paper/2405.10552/) (2024)
- [Inpainting the Gaps: A Novel Framework for Evaluating Explanation Methods in Vision Transformers](https://mlsanity.com/paper/2406.11534/) (2024)
- [COMIX: Compositional Explanations using Prototypes](https://mlsanity.com/paper/2501.06059/) (2025)

## Abstract (arXiv)

The field of explainable artificial intelligence (XAI) aims to uncover the inner workings of complex deep neural models. While being crucial for safety-critical domains, XAI inherently lacks ground-truth explanations, making its automatic evaluation an unsolved problem. We address this challenge by proposing a novel synthetic vision dataset, named FunnyBirds, and accompanying automatic evaluation protocols. Our dataset allows performing semantically meaningful image interventions, e.g., removing individual object parts, which has three important implications. First, it enables analyzing explanations on a part level, which is closer to human comprehension than existing methods that evaluate on a pixel level. Second, by comparing the model output for inputs with removed parts, we can estimate ground-truth part importances that should be reflected in the explanations. Third, by mapping individual explanations into a common space of part importances, we can analyze a variety of different explanation types in a single common framework. Using our tools, we report results for 24 different combinations of neural models and XAI methods, demonstrating the strengths and weaknesses of the assessed methods in a fully automatic and systematic manner.