# IKUN: Initialization to Keep snn training and generalization great with sUrrogate-stable variaNce

arXiv: https://arxiv.org/abs/2411.18250 · HTML: https://mlsanity.com/paper/2411.18250/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks](https://mlsanity.com/paper/1805.07866/) (2018)
- [Training Deep Spiking Neural Networks](https://mlsanity.com/paper/2006.04436/) (2020)
- [Identifying Generalization Properties in Neural Networks](https://mlsanity.com/paper/1809.07402/) (2018)
- [Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks](https://mlsanity.com/paper/2404.14964/) (2024)

## Abstract (arXiv)

Weight initialization significantly impacts the convergence and performance of neural networks. While traditional methods like Xavier and Kaiming initialization are widely used, they often fall short for spiking neural networks (SNNs), which have distinct requirements compared to artificial neural networks (ANNs). To address this, we introduce IKUN, a variance-stabilizing initialization method integrated with surrogate gradient functions, specifically designed for SNNs. IKUN stabilizes signal propagation, accelerates convergence, and enhances generalization. Experiments show IKUN improves training efficiency by up to 50%, achieving 95% training accuracy and 91% generalization accuracy. Hessian analysis reveals that IKUN-trained models converge to flatter minima, characterized by Hessian eigenvalues near zero on the positive side, promoting better generalization. The method is open-sourced for further exploration: hrefhttps://github.com/MaeChd/SurrogateVarStabehttps://github.com/MaeChd/SurrogateVarStabe.