ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum
Not on the map. Why?
Abstract (arXiv)
Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 5 Jul 2026.
Builds on
29 of 59 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2019On the Convergence of Adam and Beyondcited by 834
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2017Spectrally-normalized margin bounds for neural networkscited by 681
17 more
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2018Riemannian Adaptive Optimization Methodscited by 187
- 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
- 2015All you need is a good initcited by 124
- 2020Geoopt: Riemannian Optimization in PyTorchcited by 109
- 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
- 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
- 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
- 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
- 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
- 2024Scalable Optimization in the Modular Normcited by 64
- 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
- 2017Gated Orthogonal Recurrent Units: On Learning to Forgetcited by 31
- 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
Built on by
Not cited yet.