mlsanityData to 7 Oct 2026

Trivializations for Gradient-Based Optimization on Manifolds

arXiv 1909.09501 ↗First posted 20 Sep 2019cs.LG stat.MLcited by 78

Not on the map. Why?

Abstract (arXiv)

We introduce a framework to study the transformation of problems with manifold constraints into unconstrained problems through parametrizations in terms of a Euclidean space. We call these parametrizations "trivializations". We prove conditions under which a trivialization is sound in the context of gradient-based optimization and we show how two large families of trivializations have overall favorable properties, but also suffer from a performance issue. We then introduce "dynamic trivializations", which solve this problem, and we show how these form a family of optimization methods that lie between trivializations and Riemannian gradient descent, and combine the benefits of both of them. We then show how to implement these two families of trivializations in practice for different matrix manifolds. To this end, we prove a formula for the gradient of the exponential of matrices, which can be of practical interest on its own. Finally, we show how dynamic trivializations improve the performance of existing methods on standard tasks designed to test long-term memory within neural networks.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 8; 2023: 15; 2024: 10; 2025: 16; 2026: 10.

78 papers.

Sort:

By area:

  1. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  2. 2021Modeling Irregular Time Series with Continuous Recurrent Unitscited by 101
  3. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  4. 2021Scaling Scaling Laws with Board Gamescited by 62
  5. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  6. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  7. 2021Long Expressive Memory for Sequence Modelingcited by 41
  8. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  9. 2023Deep networks for system identification: a Surveycited by 35
  10. 2021Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flowscited by 30
  11. 2023RAYEN: Imposition of Hard Convex Constraints on Neural Networkscited by 30
  12. 2022Contrastive Laplacian Eigenmapscited by 24
38 more
  1. 2024Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groupscited by 23
  2. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  3. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  4. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
  5. 2023Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networkscited by 17
  6. 2023Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraintscited by 15
  7. 2021Nested Hyperbolic Spaces for Dimensionality Reduction and Hyperbolic NN Designcited by 14
  8. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  9. 2021An Operator Theoretic Approach for Analyzing Sequence Neural Networkscited by 12
  10. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  11. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  12. 2022O-ViT: Orthogonal Vision Transformercited by 9
  13. 2024Amortized Control of Continuous State Space Feynman-Kac Model for Irregular Time Seriescited by 9
  14. 2022Path Development Network with Finite-dimensional Lie Group Representationcited by 8
  15. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  16. 2023Bootstrapping Parallel Anchors for Relative Representationscited by 7
  17. 2023A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalizationcited by 7
  18. 2024Learning Hamiltonian neural Koopman operator and simultaneously sustaining and discovering conservation lawcited by 7
  19. 2022Multi-View Independent Component Analysis with Shared and Individual Sourcescited by 6
  20. 2023Uncovering Challenges of Solving the Continuous Gromov-Wasserstein Problemcited by 6
  21. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  22. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  23. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  24. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  25. 2023Unsupervised Interpretable Basis Extraction for Concept-Based Visual Explanationscited by 5
  26. 2025SPD Matrix Learning for Neuroimaging Analysis: Perspectives, Methods, and Challengescited by 5
  27. 2025How to Square Tensor Networks and Circuits Without Squaring Themcited by 5
  28. 2026Neural Networks on Symmetric Spaces of Noncompact Typecited by 5
  29. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4
  30. 2021Deep Decomposition for Stochastic Normal-Abnormal Transportcited by 4
  31. 2024Riemannian coordinate descent algorithms on matrix manifoldscited by 4
  32. 2024Fast and Stable Riemannian Metrics on SPD Manifolds via Cholesky Product Geometrycited by 4
  33. 2025Position: Adopt Constraints Over Fixed Penalties in Deep Learningcited by 4
  34. 2025Efficient Robust Conformal Prediction via Lipschitz-Bounded Networkscited by 4
  35. 2026JPmHC Dynamical Isometry via Orthogonal Hyper-Connectionscited by 4
  36. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  37. 2022Geometry-Aware Supertagging with Heterogeneous Dynamic Convolutionscited by 3
  38. 2023Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representationcited by 3

50 most cited of 78 All 78 (JSON)