mlsanityData to 7 Oct 2026

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

arXiv 1602.05897 ↗First posted 18 Feb 2016cs.LG cs.AI cs.CCcited by 138

Not on the map. Why?

Abstract (arXiv)

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 44; 2022: 26; 2023: 28; 2024: 18; 2025: 12; 2026: 10.

138 papers.

Sort:

By area:

  1. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  2. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  3. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  4. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  5. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  6. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  7. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  8. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  9. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  10. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  11. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  12. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
38 more
  1. 2021High Quality Segmentation for Ultra High-resolution Imagescited by 31
  2. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  3. 2021Properties of the After Kernelcited by 27
  4. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  5. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  6. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  7. 2025CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agentcited by 22
  8. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  9. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  10. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  11. 2022Infinite Recommendation Networks: A Data-Centric Approachcited by 20
  12. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  13. 2023Six Lectures on Linearized Neural Networkscited by 19
  14. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
  15. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 18
  16. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  17. 2023A Rainbow in Deep Network Black Boxescited by 18
  18. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  19. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  20. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  21. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  22. 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
  23. 2021Untrained Graph Neural Networks for Denoisingcited by 15
  24. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  25. 2021Convergence of Deep ReLU Networkscited by 14
  26. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  27. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  28. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  29. 2021Fundamental Tradeoffs in Distributionally Adversarial Trainingcited by 12
  30. 2021Reverse Engineering the Neural Tangent Kernelcited by 12
  31. 2022Wide and Deep Neural Networks Achieve Optimality for Classificationcited by 12
  32. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  33. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  34. 2021Implicit Bias of Linear RNNscited by 11
  35. 2023Efficient SGD Neural Network Training via Sublinear Activated Neuron Identificationcited by 11
  36. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 10
  37. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  38. 2023A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networkscited by 10

50 most cited of 138 All 138 (JSON)