mlsanityData to 7 Oct 2026

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

arXiv 1605.06743 ↗First posted 22 May 2016cs.NE cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network's pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 9; 2023: 9; 2024: 7; 2025: 2; 2026: 5.

53 papers.

Sort:

By area:

  1. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  2. 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
  3. 2021Towards Robust Vision Transformercited by 137
  4. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  5. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  6. 2021Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learningcited by 56
  7. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  8. 2021Implicit Regularization in Tensor Factorizationcited by 43
  9. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  10. 2021Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)cited by 33
  11. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  12. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
38 more
  1. 2021iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Imagescited by 27
  2. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  3. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  4. 2021MC-LSTM: Mass-Conserving LSTMcited by 17
  5. 2022YOLOPose: Transformer-based Multi-Object 6D Pose Estimation using Keypoint Regressioncited by 17
  6. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16
  7. 2021A Random CNN Sees Objects: One Inductive Bias of CNN and Its Applicationscited by 14
  8. 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
  9. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  10. 2022On the Ability of Graph Neural Networks to Model Interactions Between Verticescited by 12
  11. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 12
  12. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  13. 2021Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-trainingcited by 8
  14. 2022Contour-Aware Equipotential Learning for Semantic Segmentationcited by 6
  15. 2021Entangled q-Convolutional Neural Netscited by 4
  16. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  17. 2023Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalizationcited by 4
  18. 2023Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalizationcited by 4
  19. 2022Forecasting through deep learning and modal decomposition in two-phase concentric jetscited by 3
  20. 2023YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimationcited by 3
  21. 2023Exponential Quantum Communication Advantage in Distributed Inference and Learningcited by 3
  22. 2024A Tensor Decomposition Perspective on Second-order RNNscited by 3
  23. 2022Frequency Dropout: Feature-Level Regularization via Randomized Filteringcited by 2
  24. 2024Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentationcited by 2
  25. 2021Training Deep Normalizing Flow Models in Highly Incomplete Data Scenarios with Prior Regularizationcited by 1
  26. 2023Scene Structure Guidance Network: Unfolding Graph Partitioning into Pixel-Wise Feature Learningcited by 1
  27. 2024Anatomy of Neural Language Modelscited by 1
  28. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  29. 2024STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processingcited by 1
  30. 2026MuViS: Multimodal Virtual Sensing Benchmarkcited by 1
  31. 2021Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modellingcited by 0
  32. 2021Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?cited by 0
  33. 2022Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problemscited by 0
  34. 2022A lightweight Transformer-based model for fish landmark detectioncited by 0
  35. 2023Manifold Regularization for Memory-Efficient Training of Deep Neural Networkscited by 0
  36. 2023Efficient Multi-Object Pose Estimation using Multi-Resolution Deformable Attention and Query Aggregationcited by 0
  37. 2024Knowledge distillation to effectively attain both region-of-interest and global semantics from an image where multiple objects appearcited by 0
  38. 2026A New Convergence Analysis of Plug-and-Play Proximal Gradient Descent Under Prior Mismatchcited by 0

50 most cited of 53 All 53 (JSON)