mlsanityData to 14 Oct 2026

All you need is a good init

arXiv 1511.06422 ↗First posted 19 Nov 2015cs.LGcited by 124

Not on the map. Why?

Abstract (arXiv)

Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 25; 2023: 22; 2024: 16; 2025: 16; 2026: 13.

124 papers.

Sort:

By area:

  1. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  2. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  3. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  4. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  5. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 64
  6. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  7. 2021Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifierscited by 57
  8. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  9. 2021A Survey on Green Deep Learningcited by 52
  10. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  11. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  12. 2021Orthogonal Projection Losscited by 44
38 more
  1. 2022Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 42
  2. 2023Initializing Models with Larger Onescited by 39
  3. 2021KNAS: Green Neural Architecture Searchcited by 33
  4. 2022Random vector functional link network: recent developments, applications, and future directionscited by 33
  5. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  6. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  7. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  8. 2021An evidential classifier based on Dempster-Shafer theory and deep learningcited by 18
  9. 2023Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruningcited by 16
  10. 2023A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networkscited by 16
  11. 2023Transformer as Linear Expansion of Learngenecited by 13
  12. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  13. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  14. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  15. 2024Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Modelscited by 11
  16. 2025Initialization Schemes for Kolmogorov-Arnold Networks: An Empirical Studycited by 11
  17. 2023Are GATs Out of Balance?cited by 10
  18. 2021A Hierarchical Conditional Random Field-based Attention Mechanism Approach for Gastric Histopathology Image Classificationcited by 9
  19. 2021Neuron Campaign for Initialization Guided by Information Bottleneck Theorycited by 9
  20. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 9
  21. 2022Statistical guarantees for sparse deep learningcited by 9
  22. 2024Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Modelscited by 9
  23. 2025RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptationcited by 9
  24. 2022Batch Normalization Explainedcited by 8
  25. 2023Best Practices for 2-Body Pose Forecastingcited by 8
  26. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  27. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  28. 2022Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularizationcited by 7
  29. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  30. 2021Max-Affine Spline Insights Into Deep Network Pruningcited by 5
  31. 2021Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networkscited by 5
  32. 2021AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networkscited by 5
  33. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  34. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  35. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  36. 2023Multi-Objective Optimization for Sparse Deep Multi-Task Learningcited by 5
  37. 2024Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysiscited by 5
  38. 2024Residual Kolmogorov-Arnold Network for Enhanced Deep Learningcited by 5

50 most cited of 124 All 124 (JSON)