mlsanityData to 7 Oct 2026

A Rainbow in Deep Network Black Boxes

arXiv 2305.18512 ↗First posted 29 May 2023cs.LG cs.CV eess.SPcited by 18

Not on the map. Why?

Abstract (arXiv)

A central question in deep learning is to understand the functions learned by deep networks. What is their approximation class? Do the learned weights and representations depend on initialization? Previous empirical work has evidenced that kernels defined by network activations are similar across initializations. For shallow networks, this has been theoretically studied with random feature models, but an extension to deep networks has remained elusive. Here, we provide a deep extension of such random feature models, which we call the rainbow model. We prove that rainbow networks define deterministic (hierarchical) kernels in the infinite-width limit. The resulting functions thus belong to a data-dependent RKHS which does not depend on the weight randomness. We also verify numerically our modeling assumptions on deep CNNs trained on image classification tasks, and show that the trained networks approximately satisfy the rainbow hypothesis. In particular, rainbow networks sampled from the corresponding random feature model achieve similar performance as the trained networks. Our results highlight the central role played by the covariances of network weights at each layer, which are observed to be low-rank as a result of feature learning.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 69 of its 129 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

69 of 129 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  10. 2016Group Equivariant Convolutional Networkscited by 1,154
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
38 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 671
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  9. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  11. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  13. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  14. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  15. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  16. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  17. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  18. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  19. 2013Predicting Parameters in Deep Learningcited by 223
  20. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  21. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  22. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  23. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  24. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  25. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  26. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  27. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  28. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  29. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  30. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  31. 2022A Solvable Model of Neural Scaling Lawscited by 101
  32. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  33. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 94
  34. 2019Robust and interpretable blind image denoising via bias-free convolutional neural networkscited by 87
  35. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  36. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  37. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  38. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 7; 2026: 3.

18 papers.

Sort:

  1. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  2. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  3. 2025The Universal Weight Subspace Hypothesiscited by 22
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  6. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  7. 2025Large Language Models and Emergence: A Complex Systems Perspectivecited by 7
  8. 2024On the universality of neural encodings in CNNscited by 4
  9. 2025Connecting Neural Models Latent Geometries with Relative Geodesic Representationscited by 4
  10. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  11. 2025Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrizationcited by 3
  12. 2024Spring-block theory of feature learning in deep neural networkscited by 2
6 more
  1. 2025The Effect of Label Noise on the Information Content of Neural Representationscited by 2
  2. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  3. 2024Latent Functional Maps: a spectral framework for representation alignmentcited by 1
  4. 2025On the Neural Feature Ansatz for Deep Neural Networkscited by 1
  5. 2026Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentationcited by 0
  6. 2026Localizing Transfer Between Memorization Taskscited by 0