mlsanityData to 14 Oct 2026

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks

arXiv 2305.15912 ↗First posted 25 May 2023cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP), a novel neural network parameterization technique that effectively separates the radial and angular components of weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 50 of its 71 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

50 of 71 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Wide Residual Networkscited by 3,070
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2018Group Normalizationcited by 1,524
  11. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  12. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
38 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  2. 2017Deep Neural Networks as Gaussian Processescited by 622
  3. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  5. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  6. 2018How Does Batch Normalization Help Optimization?cited by 357
  7. 2016On the Expressive Power of Deep Neural Networkscited by 306
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  9. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  12. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  13. 2019Complexity of Linear Regions in Deep Networkscited by 167
  14. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  15. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  16. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  17. 2020Optimal Client Sampling for Federated Learningcited by 124
  18. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  19. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  20. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  21. 2019Reverse-Engineering Deep ReLU Networkscited by 83
  22. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  23. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  24. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  25. 2017Deep Hyperspherical Learningcited by 55
  26. 2020Orthogonal Over-Parameterized Trainingcited by 44
  27. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  28. 2018Differentiable Learning-to-Normalize via Switchable Normalizationcited by 33
  29. 2019Online Normalization for Training Neural Networkscited by 29
  30. 2019Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noisecited by 28
  31. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  32. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  33. 2020Regularized linear autoencoders recover the principal components, eventuallycited by 26
  34. 2023Joint Training of Deep Ensembles Fails Due to Learner Collusioncited by 23
  35. 2018A comparison of deep networks with ReLU activation function and linear spline-type methodscited by 18
  36. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
  37. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  38. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2023Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Trainingcited by 4
  2. 2025Structured Transformations for Stable and Interpretable Neural Computationcited by 2
  3. 2025Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Datacited by 0
  4. 2026A Composite Activation Function for Learning Stable Binary Representationscited by 0
  5. 2026Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulascited by 0