# Deep Convolutional Networks are Hierarchical Kernel Machines

arXiv: https://arxiv.org/abs/1508.01084 · HTML: https://mlsanity.com/paper/1508.01084/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2015.

## Built on by (most cited first, 9 of 9)

- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernel](https://mlsanity.com/paper/2103.11558/) (2021)
- [Multi-Resolution Online Deterministic Annealing: A Hierarchical and Progressive Learning Architecture](https://mlsanity.com/paper/2212.08189/) (2022)
- [On the Power of Decision Trees in Auto-Regressive Language Modeling](https://mlsanity.com/paper/2409.19150/) (2024)
- [Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach](https://mlsanity.com/paper/2112.02256/) (2021)
- [Stability Analysis of Equivariant Convolutional Representations Through The Lens of Equivariant Multi-layered CKNs](https://mlsanity.com/paper/2408.04277/) (2024)
- [A resource-efficient model for deep kernel learning](https://mlsanity.com/paper/2410.09926/) (2024)

## Abstract (arXiv)

In i-theory a typical layer of a hierarchical architecture consists of HW modules pooling the dot products of the inputs to the layer with the transformations of a few templates under a group. Such layers include as special cases the convolutional layers of Deep Convolutional Networks (DCNs) as well as the non-convolutional layers (when the group contains only the identity). Rectifying nonlinearities -- which are used by present-day DCNs -- are one of the several nonlinearities admitted by i-theory for the HW module. We discuss here the equivalence between group averages of linear combinations of rectifying nonlinearities and an associated kernel. This property implies that present-day DCNs can be exactly equivalent to a hierarchy of kernel machines with pooling and non-pooling layers. Finally, we describe a conjecture for theoretically understanding hierarchies of such modules. A main consequence of the conjecture is that hierarchies of trained HW modules minimize memory requirements while computing a selective and invariant representation.