mlsanityData to 7 Oct 2026

Towards Theoretically Inspired Neural Initialization Optimization

arXiv 2210.05956 ↗First posted 12 Oct 2022cs.LG cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters. In the domain of neural initialization, however, similar automated techniques have rarely been studied. Most existing initialization methods are handcrafted and highly dependent on specific architectures. In this paper, we propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. Specifically, GradCosine is the cosine similarity of sample-wise gradients with respect to the initialized parameters. By analyzing the sample-wise optimization landscape, we show that both the training and test performance of a network can be improved by maximizing GradCosine under gradient norm constraint. Based on this observation, we further propose the neural initialization optimization (NIO) algorithm. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost compared with the training time. With NIO, we improve the classification performance of a variety of neural architectures on CIFAR-10, CIFAR-100, and ImageNet. Moreover, we find that our method can even help to train large vision Transformer architecture without warmup.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

49 of 76 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
37 more
  1. 2016Wide Residual Networkscited by 3,057
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2016An overview of gradient descent optimization algorithmscited by 1,021
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  11. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  12. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  13. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  14. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  15. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  16. 2020Neural Architecture Search without Trainingcited by 241
  17. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  18. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  19. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  20. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  21. 2015All you need is a good initcited by 124
  22. 2018Searching for Efficient Multi-Scale Architectures for Dense Image Predictioncited by 96
  23. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  24. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  25. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  26. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  27. 2017Learning ReLUs via Gradient Descentcited by 54
  28. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  29. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  30. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  31. 2020ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Codingcited by 36
  32. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  33. 2021Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Searchcited by 27
  34. 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
  35. 2018Optimization Algorithm Inspired Deep Neural Network Structure Designcited by 14
  36. 2021Neural Architecture Search as Program Transformation Explorationcited by 8
  37. 2021Data-driven Weight Initialization with Sylvester Solverscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 4.

11 papers.

  1. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  2. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  3. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 8
  4. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  5. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  6. 2024Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikescited by 3
  7. 2025Optimization-Inspired Few-Shot Adaptation for Large Language Modelscited by 3
  8. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  9. 2024LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameterscited by 0
  10. 2024Advancing Neural Network Performance through Emergence-Promoting Initialization Schemecited by 0
  11. 2025Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgettingcited by 0