# Fast Differentiable SVD on GPU via Polar Decomposition

arXiv: https://arxiv.org/abs/2609.32505 · HTML: https://mlsanity.com/paper/2609.32505/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization](https://mlsanity.com/paper/1904.06836/) (2019)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Domain Generalization via Nuclear Norm Regularization](https://mlsanity.com/paper/2303.07527/) (2023)
- [Efficient GPU implementation of randomized SVD and its applications](https://mlsanity.com/paper/2110.03423/) (2021)
- [COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation](https://mlsanity.com/paper/2507.07580/) (2025)
- [Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problems](https://mlsanity.com/paper/2411.14141/) (2024)
- [Matrix Calculus (for Machine Learning and Beyond)](https://mlsanity.com/paper/2501.14787/) (2025)

## Abstract (arXiv)

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a 2× speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX: https://github.com/fallnlove/canssvd.