# Padding Aware Neurons

arXiv: https://arxiv.org/abs/2309.08048 · HTML: https://mlsanity.com/paper/2309.08048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Mind the Pad -- CNNs can Develop Blind Spots](https://mlsanity.com/paper/2010.02178/) (2020)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [Convolution with even-sized kernels and symmetric padding](https://mlsanity.com/paper/1903.08385/) (2019)
- [Context-aware Padding for Semantic Segmentation](https://mlsanity.com/paper/2109.07854/) (2021)

## Abstract (arXiv)

Convolutional layers are a fundamental component of most image-related models. These layers often implement by default a static padding policy (eg zero padding), to control the scale of the internal representations, and to allow kernel activations centered on the border regions. In this work we identify Padding Aware Neurons (PANs), a type of filter that is found in most (if not all) convolutional models trained with static padding. PANs focus on the characterization and recognition of input border location, introducing a spatial inductive bias into the model (e.g., how close to the input's border a pattern typically is). We propose a method to identify PANs through their activations, and explore their presence in several popular pre-trained models, finding PANs on all models explored, from dozens to hundreds. We discuss and illustrate different types of PANs, their kernels and behaviour. To understand their relevance, we test their impact on model performance, and find padding and PANs to induce strong and characteristic biases in the data. Finally, we discuss whether or not PANs are desirable, as well as the potential side effects of their presence in the context of model performance, generalisation, efficiency and safety.