mlsanityData to 14 Oct 2026

Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

arXiv 2610.20069 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + λz 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

23 of 33 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  11. 2016Language Modeling with Gated Convolutional Networkscited by 702
  12. 2024KAN: Kolmogorov-Arnold Networkscited by 649
11 more
  1. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  2. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  3. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  4. 2015ParseNet: Looking Wider to See Bettercited by 177
  5. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  6. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  7. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 66
  8. 2017Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networkscited by 59
  9. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  10. 2020Spherical Perspective on Learning with Normalization Layerscited by 10
  11. 2021The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometrycited by 2

Built on by

Not cited yet.