# Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

arXiv: https://arxiv.org/abs/2101.12322 · HTML: https://mlsanity.com/paper/2101.12322/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 90 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)

## Built on by (most cited first, 26 of 26)

- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework](https://mlsanity.com/paper/2503.14880/) (2025)
- [Deep Generative Data Assimilation in Multimodal Setting](https://mlsanity.com/paper/2404.06665/) (2024)
- [High-fidelity GAN Inversion with Padding Space](https://mlsanity.com/paper/2203.11105/) (2022)
- [Leveraging Equivariant Features for Absolute Pose Regression](https://mlsanity.com/paper/2204.02163/) (2022)
- [Counting Varying Density Crowds Through Density Guided Adaptive Selection CNN and Transformer Estimation](https://mlsanity.com/paper/2206.10075/) (2022)
- [Mitigating the Bias of Centered Objects in Common Datasets](https://mlsanity.com/paper/2112.09195/) (2021)
- [Zone Evaluation: Revealing Spatial Bias in Object Detection](https://mlsanity.com/paper/2310.13215/) (2023)
- [Padding Module: Learning the Padding in Deep Neural Networks](https://mlsanity.com/paper/2301.04608/) (2023)
- [Unveiling The Mask of Position-Information Pattern Through the Mist of Image Features](https://mlsanity.com/paper/2206.01202/) (2022)
- [De-coupling and De-positioning Dense Self-supervised Learning](https://mlsanity.com/paper/2303.16947/) (2023)
- [PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition](https://mlsanity.com/paper/2407.02934/) (2024)
- [Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attention](https://mlsanity.com/paper/2411.19261/) (2024)
- [Multi Kernel Positional Embedding ConvNeXt for Polyp Segmentation](https://mlsanity.com/paper/2301.06673/) (2023)
- [Robust and Explainable Fine-Grained Visual Classification with Transfer Learning: A Dual-Carriageway Framework](https://mlsanity.com/paper/2405.05853/) (2024)
- [LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models](https://mlsanity.com/paper/2601.18513/) (2026)
- [Deep Learning Empowered Type-II Codebook: New Paradigm for Enhancing CSI Feedback](https://mlsanity.com/paper/2305.08081/) (2023)
- [Imperceptible Adversarial Attack on Deep Neural Networks from Image Boundary](https://mlsanity.com/paper/2308.15344/) (2023)
- [Padding-free Convolution based on Preservation of Differential Characteristics of Kernels](https://mlsanity.com/paper/2309.06370/) (2023)
- [An Investigation on The Position Encoding in Vision-Based Dynamics Prediction](https://mlsanity.com/paper/2408.15201/) (2024)
- [Improvements to SDXL in NovelAI Diffusion V3](https://mlsanity.com/paper/2409.15997/) (2024)
- [Auto-encoder model for faster generation of effective one-body gravitational waveform approximations](https://mlsanity.com/paper/2511.12642/) (2025)
- [Lane detection with Position Embedding](https://mlsanity.com/paper/2203.12301/) (2022)
- [Rethinking Query-Key Pairwise Interactions in Vision Transformers](https://mlsanity.com/paper/2207.00188/) (2022)
- [Improving Translation Invariance in Convolutional Neural Networks with Peripheral Prediction Padding](https://mlsanity.com/paper/2307.07725/) (2023)
- [ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model](https://mlsanity.com/paper/2408.15548/) (2024)

## Abstract (arXiv)

In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. In this paper, we first test this hypothesis and reveal that a surprising degree of absolute position information is encoded in commonly used CNNs. We show that zero padding drives CNNs to encode position information in their internal representations, while a lack of padding precludes position encoding. This gives rise to deeper questions about the role of position information in CNNs: (i) What boundary heuristics enable optimal position encoding for downstream tasks?; (ii) Does position encoding affect the learning of semantic representations?; (iii) Does position encoding always improve performance? To provide answers, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We design novel tasks which allow us to quantify boundary effects as a function of the distance to the border. Numerous semantic objectives reveal the effect of the border on semantic representations. Finally, we demonstrate the implications of these findings on multiple real-world tasks to show that position information can both help or hurt performance.