# Towards Improved Input Masking for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2211.14646 · HTML: https://mlsanity.com/paper/2211.14646/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [(De)Randomized Smoothing for Certifiable Defense against Patch Attacks](https://mlsanity.com/paper/2002.10733/) (2020)
- [Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detection](https://mlsanity.com/paper/2112.04532/) (2021)
- [MaskTune: Mitigating Spurious Correlations by Forcing to Explore](https://mlsanity.com/paper/2210.00055/) (2022)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Certified Patch Robustness via Smoothed Vision Transformers](https://mlsanity.com/paper/2110.07719/) (2021)
- [Are Vision Transformers Robust to Patch Perturbations?](https://mlsanity.com/paper/2111.10659/) (2021)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)

## Built on by (most cited first, 8 of 8)

- [Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP](https://mlsanity.com/paper/2406.01583/) (2024)
- [Towards Human-Understandable Multi-Dimensional Concept Discovery](https://mlsanity.com/paper/2503.18629/) (2025)
- [Masked Capsule Autoencoders](https://mlsanity.com/paper/2403.04724/) (2024)
- [Missingness Bias Calibration in Feature Attribution Explanations](https://mlsanity.com/paper/2603.04831/) (2026)
- [Token-Based Detection of Spurious Correlations in Vision Transformers](https://mlsanity.com/paper/2509.04009/) (2025)
- [Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI](https://mlsanity.com/paper/2510.01038/) (2025)
- [Signal or Noise? Auditing Rotation-Induced Saliency Drift in Medical and Aerial Imaging](https://mlsanity.com/paper/2609.02224/) (2026)
- [ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution](https://mlsanity.com/paper/2610.04605/) (2026)

## Abstract (arXiv)

The ability to remove features from the input of machine learning models is very important to understand and interpret model predictions. However, this is non-trivial for vision models since masking out parts of the input image typically causes large distribution shifts. This is because the baseline color used for masking (typically grey or black) is out of distribution. Furthermore, the shape of the mask itself can contain unwanted signals which can be used by the model for its predictions. Recently, there has been some progress in mitigating this issue (called missingness bias) in image masking for vision transformers. In this work, we propose a new masking method for CNNs we call layer masking in which the missingness bias caused by masking is reduced to a large extent. Intuitively, layer masking applies a mask to intermediate activation maps so that the model only processes the unmasked input. We show that our method (i) is able to eliminate or minimize the influence of the mask shape or color on the output of the model, and (ii) is much better than replacing the masked region by black or grey for input perturbation based interpretability techniques like LIME. Thus, layer masking is much less affected by missingness bias than other masking strategies. We also demonstrate how the shape of the mask may leak information about the class, thus affecting estimates of model reliance on class-relevant features derived from input masking. Furthermore, we discuss the role of data augmentation techniques for tackling this problem, and argue that they are not sufficient for preventing model reliance on mask shape. The code for this project is publicly available at https://github.com/SriramB-98/layer_masking