Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Not on the map. Why?
Abstract (arXiv)
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignments are usually fixed by layer type, without a clear criterion for selecting the appropriate geometry. We turn sign-versus-spectral assignment into a measurable, layerwise decision problem. For ideal memoryless steps, we derive a finite-batch ratio of conservative descent-lemma lower bounds for spectral and sign updates. Its closed form separates signal efficiency from a geometry-dependent global curvature factor. In the positive-signal regime, the same smoothness assumption gives two-sided bounds that determine which step has the larger conditional expected decrease when the ratio crosses explicit noise-dependent thresholds. We then derive a local directional quadratic score from gradient alignments and Hessian-vector products and use it in Generalized-Scion (G-Scion) to control layerwise routing. In a controlled input-rank experiment, including directional curvature matches the measured same-iterate ordering in all seven aggregate settings, whereas the curvature-free proxy selects the opposite geometry. On 31M GPT-2, the output score routes the output layer from spectral warmup to sign, and G-Scion improves validation loss over fixed Scion in memoryless and matched-momentum comparisons. ViT-B/16 results further show that a spectral edge assignment can outperform fixed Scion outside language modelling. These results suggest that optimizer geometry need not be hard-coded by layer type and are a first step toward understanding how it should be assigned.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
30 of 40 references.
References · 30
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2016Gaussian Error Linear Units (GELUs)cited by 3,502
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
- 2025Muon is Scalable for LLM Trainingcited by 428
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
- 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
- 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
18 more
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
- 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
- 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
- 2025NorMuon: Making Muon more efficient and scalablecited by 84
- 2025Practical Efficiency of Muon for Pretrainingcited by 80
- 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
- 2025AdaMuon: Adaptive Muon Optimizercited by 67
- 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
- 2020The Geometry of Sign Gradient Descentcited by 41
- 2025When do spectral gradient updates help in deep learning?cited by 31
- 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
- 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
- 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
- 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
- 2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
- 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
- 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
- 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 2
Built on by
Not cited yet.