mlsanityData to 7 Oct 2026

Learning from Randomly Initialized Neural Network Features

arXiv 2202.06438 ↗First posted 13 Feb 2022cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which is inherently infinite-dimensional. We conduct ablations across multiple architectures of varying sizes as well as initializations and activation functions. Our analysis suggests that certain structures that manifest in a trained model are already present at initialization. Therefore, NNPK may provide further insight into why neural networks are so effective in learning such structures.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 13 Feb 2022.

Builds on

33 of 48 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2017Deep Neural Networks as Gaussian Processescited by 617
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
21 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  2. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  4. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  5. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  6. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  7. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  8. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  9. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  10. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  11. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  12. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  13. 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
  14. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  15. 2019TriMap: Large-scale Dimensionality Reduction Using Tripletscited by 59
  16. 2020Every Model Learned by Gradient Descent Is Approximately a Kernel Machinecited by 53
  17. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  18. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  19. 2016Deep Residual Networks with Exponential Linear Unitcited by 12
  20. 2018A Gaussian Process perspective on Convolutional Neural Networkscited by 4
  21. 2020Training Deep Neural Networks Without Batch Normalizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 2; 2026: 1.

11 papers.

  1. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  2. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  3. 2023ZeroRF: Fast Sparse View 360deg Reconstruction with Zero Pretrainingcited by 24
  4. 2024Algorithmic Capabilities of Random Transformerscited by 17
  5. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  6. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 3
  7. 2025LAMP: Data-Efficient Linear Affine Weight-Space Models for Parameter-Controlled 3D Shape Generation and Extrapolationcited by 3
  8. 2025ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathologycited by 2
  9. 2022Demystifying Randomly Initialized Networks for Evaluating Generative Modelscited by 0
  10. 2022Constructing Organism Networks from Collaborative Self-Replicatorscited by 0
  11. 2026DIPBox: A Multi-scale Testing Framework for Tracking Dataset Regenerationcited by 0