# Inductive Bias of Deep Convolutional Networks through Pooling Geometry

arXiv: https://arxiv.org/abs/1605.06743 · HTML: https://mlsanity.com/paper/1605.06743/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2016.

## Built on by (most cited first, 30 of 53)

- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [Towards Robust Vision Transformer](https://mlsanity.com/paper/2105.07926/) (2021)
- [Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition](https://mlsanity.com/paper/2102.01063/) (2021)
- [Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training](https://mlsanity.com/paper/2210.07688/) (2022)
- [Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning](https://mlsanity.com/paper/2103.01315/) (2021)
- [Efficient Feature Transformations for Discriminative and Generative Continual Learning](https://mlsanity.com/paper/2103.13558/) (2021)
- [Implicit Regularization in Tensor Factorization](https://mlsanity.com/paper/2102.09972/) (2021)
- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)](https://mlsanity.com/paper/2103.01678/) (2021)
- [The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design](https://mlsanity.com/paper/2110.04541/) (2021)
- [Approximation and Learning with Deep Convolutional Models: a Kernel Perspective](https://mlsanity.com/paper/2102.10032/) (2021)
- [iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images](https://mlsanity.com/paper/2112.11325/) (2021)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [Learning with convolution and pooling operations in kernel methods](https://mlsanity.com/paper/2111.08308/) (2021)
- [MC-LSTM: Mass-Conserving LSTM](https://mlsanity.com/paper/2101.05186/) (2021)
- [YOLOPose: Transformer-based Multi-Object 6D Pose Estimation using Keypoint Regression](https://mlsanity.com/paper/2205.02536/) (2022)
- [Which transformer architecture fits my data? A vocabulary bottleneck in self-attention](https://mlsanity.com/paper/2105.03928/) (2021)
- [A Random CNN Sees Objects: One Inductive Bias of CNN and Its Applications](https://mlsanity.com/paper/2106.09259/) (2021)
- [On the Sample Complexity of Learning under Invariance and Geometric Stability](https://mlsanity.com/paper/2106.07148/) (2021)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Aligning Multimodal Representations through an Information Bottleneck](https://mlsanity.com/paper/2506.04870/) (2025)
- [On the Ability of Graph Neural Networks to Model Interactions Between Vertices](https://mlsanity.com/paper/2211.16494/) (2022)
- [The Hessian perspective into the Nature of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09088/) (2023)
- [Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training](https://mlsanity.com/paper/2112.03552/) (2021)
- [Contour-Aware Equipotential Learning for Semantic Segmentation](https://mlsanity.com/paper/2210.00223/) (2022)
- [Entangled q-Convolutional Neural Nets](https://mlsanity.com/paper/2103.11785/) (2021)
- [What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement](https://mlsanity.com/paper/2303.11249/) (2023)
- [Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization](https://mlsanity.com/paper/2308.14418/) (2023)
- [Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalization](https://mlsanity.com/paper/2311.06224/) (2023)

## Abstract (arXiv)

Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network's pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.