mlsanityData to 14 Oct 2026

Control Allocation in Neural Network Optimization: Joint Affine Control of Weight and Bias Updates

arXiv 2608.02991 ↗First posted 4 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Optimization algorithms determine not only the magnitude of a neural-network update but also how that update is distributed across parameter channels. We study whether this distribution can be treated as a controllable quantity independently of global training progress. We define operational update allocation through normalized channel energies and analyze two scalar controls: a coordinate-preconditioning exponent and an affine spectral exponent that scales the bias column of an augmented weight--bias matrix. At a frozen state, a common nonzero step-size multiplier leaves normalized allocation unchanged; the coordinate exponent yields affine pairwise log-odds with an explicit inverse; and the affine exponent induces a rank-one positive-semidefinite Gram perturbation and a logistic raw-participation law. We further separate raw affine participation, spectral gain, and the decoded physical bias update, and show that finite polynomial spectral iterations preserve singular subspaces. Same-state replay verifies the exact control laws. On a five-seed controlled benchmark, intermediate controls improve held-out and worst-group metrics, whereas excessive affine control causes underfitting. A four-task single-seed transfer study provides descriptive corroboration. These results establish instantaneous allocation control and a bounded empirical operating regime, but do not imply a task-independent generalization ordering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

13 of 17 references.

References · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2025Muon is Scalable for LLM Trainingcited by 428
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  8. 2020Directional convergence and alignment in deep learningcited by 178
  9. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  10. 2018Convergence of Gradient Descent on Separable Datacited by 110
  11. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  12. 2024Modular Duality in Deep Learningcited by 77
1 more
  1. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25

Built on by

Not cited yet.