# Missingness Bias in Model Debugging

arXiv: https://arxiv.org/abs/2204.08945 · HTML: https://mlsanity.com/paper/2204.08945/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)

## Built on by (most cited first, 30 of 38)

- [Discovering and Mitigating Visual Biases through Keyword Explanation](https://mlsanity.com/paper/2301.11104/) (2023)
- [Diagnosing and Rectifying Vision Models using Language](https://mlsanity.com/paper/2302.04269/) (2023)
- [Learning to Estimate Shapley Values with Vision Transformers](https://mlsanity.com/paper/2206.05282/) (2022)
- [A Data-Based Perspective on Transfer Learning](https://mlsanity.com/paper/2207.05739/) (2022)
- [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024)
- [Estimating Conditional Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2306.03301/) (2023)
- [GLIME: General, Stable and Local LIME Explanation](https://mlsanity.com/paper/2311.15722/) (2023)
- [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024)
- [Locality Alignment Improves Vision-Language Models](https://mlsanity.com/paper/2410.11087/) (2024)
- [Probabilistic Stability Guarantees for Feature Attributions](https://mlsanity.com/paper/2504.13787/) (2025)
- [Towards Improved Input Masking for Convolutional Neural Networks](https://mlsanity.com/paper/2211.14646/) (2022)
- [Take 5: Interpretable Image Classification with a Handful of Features](https://mlsanity.com/paper/2303.13166/) (2023)
- [SHAP-XRT: The Shapley Value Meets Conditional Independence Testing](https://mlsanity.com/paper/2207.07038/) (2022)
- [Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups](https://mlsanity.com/paper/2310.16316/) (2023)
- [Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models](https://mlsanity.com/paper/2404.02618/) (2024)
- [On the Evaluation Consistency of Attribution-based Explanations](https://mlsanity.com/paper/2407.19471/) (2024)
- [What could go wrong? Discovering and describing failure modes in computer vision](https://mlsanity.com/paper/2408.04471/) (2024)
- [T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers](https://mlsanity.com/paper/2403.04523/) (2024)
- [Layerwise Change of Knowledge in Neural Networks](https://mlsanity.com/paper/2409.08712/) (2024)
- [Towards Human-Understandable Multi-Dimensional Concept Discovery](https://mlsanity.com/paper/2503.18629/) (2025)
- [Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers](https://mlsanity.com/paper/2504.20902/) (2025)
- [Data-Centric Debugging: mitigating model failures via targeted data collection](https://mlsanity.com/paper/2211.09859/) (2022)
- [Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration](https://mlsanity.com/paper/2511.10439/) (2025)
- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)
- [Weakly Supervised Learning Significantly Reduces the Number of Labels Required for Intracranial Hemorrhage Detection on Head CT](https://mlsanity.com/paper/2211.15924/) (2022)
- [Rethinking Explanation Evaluation under the Retraining Scheme](https://mlsanity.com/paper/2511.08281/) (2025)
- [Back to the Baseline: Examining Baseline Effects on Explainability Metrics](https://mlsanity.com/paper/2512.11433/) (2025)
- [Missingness Bias Calibration in Feature Attribution Explanations](https://mlsanity.com/paper/2603.04831/) (2026)
- [Scaling Inherently Interpretable Language Models](https://mlsanity.com/paper/2608.07594/) (2026)
- [Impact of architecture on robustness and interpretability of multispectral deep neural networks](https://mlsanity.com/paper/2309.12463/) (2023)

## Abstract (arXiv)

Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice. Our code is available at https://github.com/madrylab/missingness