# Theoretical Analysis of Inductive Biases in Deep Convolutional Networks

arXiv: https://arxiv.org/abs/2305.08404 · HTML: https://mlsanity.com/paper/2305.08404/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Learning with invariances in random features and kernel models](https://mlsanity.com/paper/2102.13219/) (2021)
- [A Priori Estimates of the Population Risk for Two-layer Neural Networks](https://mlsanity.com/paper/1810.06397/) (2018)
- [Generalization bounds for deep convolutional neural networks](https://mlsanity.com/paper/1905.12600/) (2019)
- [Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?](https://mlsanity.com/paper/2010.08515/) (2020)
- [Approximation and Learning with Deep Convolutional Models: a Kernel Perspective](https://mlsanity.com/paper/2102.10032/) (2021)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks](https://mlsanity.com/paper/1607.01434/) (2016)
- [Generalization Bounds for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01487/) (2019)
- [Learning with convolution and pooling operations in kernel methods](https://mlsanity.com/paper/2111.08308/) (2021)
- [How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?](https://mlsanity.com/paper/1805.07883/) (2018)
- [Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm](https://mlsanity.com/paper/2102.12238/) (2021)
- [On the non-universality of deep learning: quantifying the cost of symmetry](https://mlsanity.com/paper/2208.03113/) (2022)
- [Approximation Properties of Deep ReLU CNNs](https://mlsanity.com/paper/2109.00190/) (2021)
- [Approximation Theory of Convolutional Architectures for Time Series Modelling](https://mlsanity.com/paper/2107.09355/) (2021)
- [Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm](https://mlsanity.com/paper/2207.04612/) (2022)
- [Approximation analysis of CNNs from a feature extraction view](https://mlsanity.com/paper/2210.09041/) (2022)

## Built on by (most cited first, 21 of 21)

- [MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning](https://mlsanity.com/paper/2404.13591/) (2024)
- [On the Inductive Bias of Stacking Towards Improving Reasoning](https://mlsanity.com/paper/2409.19044/) (2024)
- [U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models](https://mlsanity.com/paper/2404.18444/) (2024)
- [Aligning Multimodal Representations through an Information Bottleneck](https://mlsanity.com/paper/2506.04870/) (2025)
- [Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning](https://mlsanity.com/paper/2402.08010/) (2024)
- [Role of Locality and Weight Sharing in Image-Based Tasks: A Sample Complexity Separation between CNNs, LCNs, and FCNs](https://mlsanity.com/paper/2403.15707/) (2024)
- [Towards Long-Range ENSO Prediction with an Explainable Deep Learning Model](https://mlsanity.com/paper/2503.19502/) (2025)
- [GAMBAS: Generalised-Hilbert Mamba for Super-resolution of Paediatric Ultra-Low-Field MRI](https://mlsanity.com/paper/2504.04523/) (2025)
- [Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentation](https://mlsanity.com/paper/2401.10373/) (2024)
- [Learning Using a Single Forward Pass](https://mlsanity.com/paper/2402.09769/) (2024)
- [STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing](https://mlsanity.com/paper/2411.10198/) (2024)
- [Field-Space Attention for Structure-Preserving Earth System Transformers](https://mlsanity.com/paper/2512.20350/) (2025)
- [Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability](https://mlsanity.com/paper/2603.04807/) (2026)
- [From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion](https://mlsanity.com/paper/2606.12303/) (2026)
- [Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization](https://mlsanity.com/paper/2408.00923/) (2024)
- [Barriers to Complexity-Theoretic Proofs that "AGI" Using Machine Learning is Impossible](https://mlsanity.com/paper/2411.06498/) (2024)
- [Compressing Quaternion Convolutional Neural Networks for Audio Classification](https://mlsanity.com/paper/2510.21388/) (2025)
- [Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks](https://mlsanity.com/paper/2512.12898/) (2025)
- [Rethinking Transfer Learning for Industrial Inspection: DINOv3 vs. ImageNet Pretraining Across RGB and X-ray Tasks](https://mlsanity.com/paper/2605.23472/) (2026)
- [GIBLy: Improving 3D Semantic Segmentation through an Architecture-Agnostic Lightweight Geometric Inductive Bias Layer](https://mlsanity.com/paper/2605.24243/) (2026)
- [Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networks](https://mlsanity.com/paper/2607.25200/) (2026)

## Abstract (arXiv)

In this paper, we provide a theoretical analysis of the inductive biases in convolutional neural networks (CNNs). We start by examining the universality of CNNs, i.e., the ability to approximate any continuous functions. We prove that a depth of O(log d) suffices for deep CNNs to achieve this universality, where d in the input dimension. Additionally, we establish that learning sparse functions with CNNs requires only widetildeO(log2d) samples, indicating that deep CNNs can efficiently capture em long-range sparse correlations. These results are made possible through a novel combination of the multichanneling and downsampling when increasing the network depth. We also delve into the distinct roles of weight sharing and locality in CNNs. To this end, we compare the performance of CNNs, locally-connected networks (LCNs), and fully-connected networks (FCNs) on a simple regression task, where LCNs can be viewed as CNNs without weight sharing. On the one hand, we prove that LCNs require Omega(d) samples while CNNs need only widetildeO(log2d) samples, highlighting the critical role of weight sharing. On the other hand, we prove that FCNs require Omega(d2) samples, whereas LCNs need only widetildeO(d) samples, underscoring the importance of locality. These provable separations quantify the difference between the two biases, and the major observation behind our proof is that weight sharing and locality break different symmetries in the learning process.