mlsanityData to 14 Oct 2026

Fast Differentiable SVD on GPU via Polar Decomposition

arXiv 2609.32505 ↗First posted 26 Sep 2026math.NA cs.LG cs.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a 2× speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX: https://github.com/fallnlove/canssvd.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 52 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

16 of 52 references.

References · 16

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2015Automatic differentiation in machine learning: a surveycited by 855
  5. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  6. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  7. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  8. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  9. 2019Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalizationcited by 42
  10. 2021Robust Differentiable SVDcited by 19
  11. 2022Fast Differentiable Matrix Square Rootcited by 14
  12. 2023Domain Generalization via Nuclear Norm Regularizationcited by 12
4 more
  1. 2021Efficient GPU implementation of randomized SVD and its applicationscited by 5
  2. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
  3. 2024Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problemscited by 3
  4. 2025Matrix Calculus (for Machine Learning and Beyond)cited by 1

Built on by

Not cited yet.