# Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

arXiv: https://arxiv.org/abs/1504.08291 · HTML: https://mlsanity.com/paper/1504.08291/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2015.

## Built on by (most cited first, 30 of 41)

- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [When the Curious Abandon Honesty: Federated Learning Is Not Private](https://mlsanity.com/paper/2112.02918/) (2021)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Causality-inspired Single-source Domain Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2111.12525/) (2021)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [Constructing coarse-scale bifurcation diagrams from spatio-temporal observations of microscopic simulations: A parsimonious machine learning approach](https://mlsanity.com/paper/2201.13323/) (2022)
- [Sampling weights of deep neural networks](https://mlsanity.com/paper/2306.16830/) (2023)
- [Deep ReLU Networks Preserve Expected Length](https://mlsanity.com/paper/2102.10492/) (2021)
- [Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregation](https://mlsanity.com/paper/2301.04017/) (2023)
- [The Separation Capacity of Random Neural Networks](https://mlsanity.com/paper/2108.00207/) (2021)
- [Efficient and Private Federated Learning with Partially Trainable Networks](https://mlsanity.com/paper/2110.03450/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey](https://mlsanity.com/paper/2108.04172/) (2021)
- [Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics](https://mlsanity.com/paper/2108.11939/) (2021)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Phase Collapse in Neural Networks](https://mlsanity.com/paper/2110.05283/) (2021)
- [Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference](https://mlsanity.com/paper/2104.06788/) (2021)
- [Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case](https://mlsanity.com/paper/2201.04543/) (2022)
- [Deep Random Projection Outlyingness for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2106.15307/) (2021)
- [Weighting and Pruning based Ensemble Deep Random Vector Functional Link Network for Tabular Data Classification](https://mlsanity.com/paper/2201.05809/) (2022)
- [Medical records condensation: a roadmap towards healthcare data democratisation](https://mlsanity.com/paper/2305.03711/) (2023)
- [Decomposed Distribution Matching in Dataset Condensation](https://mlsanity.com/paper/2412.04748/) (2024)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology](https://mlsanity.com/paper/2503.17564/) (2025)
- [Numerical Solution of Stiff ODEs with Physics-Informed RPNNs](https://mlsanity.com/paper/2108.01584/) (2021)
- [On the Role of Neural Collapse in Meta Learning Models for Few-shot Learning](https://mlsanity.com/paper/2310.00451/) (2023)
- [Foundation Model's Embedded Representations May Detect Distribution Shift](https://mlsanity.com/paper/2310.13836/) (2023)

## Abstract (arXiv)

Three important properties of a classification machinery are: (i) the system preserves the core information of the input data; (ii) the training examples convey information about unseen data; and (iii) the system is able to treat differently points from different classes. In this work we show that these fundamental properties are satisfied by the architecture of deep neural networks. We formally prove that these networks with random Gaussian weights perform a distance-preserving embedding of the data, with a special treatment for in-class and out-of-class data. Similar points at the input of the network are likely to have a similar output. The theoretical analysis of deep networks here presented exploits tools used in the compressed sensing and dictionary learning literature, thereby making a formal connection between these important topics. The derived results allow drawing conclusions on the metric learning properties of the network and their relation to its structure, as well as providing bounds on the required size of the training set such that the training examples would represent faithfully the unseen data. The results are validated with state-of-the-art trained networks.