mlsanityData to 7 Oct 2026

A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs

arXiv 2111.02155 ↗First posted 3 Nov 2021cs.LG cs.NE math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network? The celebrated Johnson--Lindenstrauss lemma answers this question for linear fully-connected neural networks (FNNs), stating that the geometry is essentially preserved. For FNNs with the ReLU activation, the angle between two inputs contracts according to a known mapping. The question for non-linear convolutional neural networks (CNNs) becomes much more intricate. To answer this question, we introduce a geometric framework. For linear CNNs, we show that the Johnson--Lindenstrauss lemma continues to hold, namely, that the angle between two inputs is preserved. For CNNs with ReLU activation, on the other hand, the behavior is richer: The angle between the outputs contracts, where the level of contraction depends on the nature of the inputs. In particular, after one layer, the geometry of natural images is essentially preserved, whereas for Gaussian correlated inputs, CNNs exhibit the same contracting behavior as FNNs with ReLU activation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

16 of 29 references.

References · 16

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  4. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  5. 2016Deep Information Propagationcited by 194
  6. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  7. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  8. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  9. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  10. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  11. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  12. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
4 more
  1. 2015Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?cited by 40
  2. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  3. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  4. 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22

Built on by

Not cited yet.