# Robust Differentiable SVD

arXiv: https://arxiv.org/abs/2104.03821 · HTML: https://mlsanity.com/paper/2104.03821/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 63 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Switchable Whitening for Deep Representation Learning](https://mlsanity.com/paper/1904.09739/) (2019)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [Iterative Normalization: Beyond Standardization towards Efficient Whitening](https://mlsanity.com/paper/1904.03441/) (2019)
- [StyleBank: An Explicit Representation for Neural Image Style Transfer](https://mlsanity.com/paper/1703.09210/) (2017)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization](https://mlsanity.com/paper/1712.01034/) (2017)
- [Diversified Texture Synthesis with Feed-forward Networks](https://mlsanity.com/paper/1703.01664/) (2017)
- [Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation](https://mlsanity.com/paper/1812.09912/) (2018)
- [Is Second-order Information Helpful for Large-scale Visual Recognition?](https://mlsanity.com/paper/1703.08050/) (2017)
- [Binge Watching: Scaling Affordance Learning from Sitcoms](https://mlsanity.com/paper/1804.03080/) (2018)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)
- [Whitening and Coloring batch transform for GANs](https://mlsanity.com/paper/1806.00420/) (2018)
- [Backpropagation-Friendly Eigendecomposition](https://mlsanity.com/paper/1906.09023/) (2019)
- [Dynamic Label Graph Matching for Unsupervised Video Re-Identification](https://mlsanity.com/paper/1709.09297/) (2017)
- [Every Smile is Unique: Landmark-Guided Diverse Smile Generation](https://mlsanity.com/paper/1802.01873/) (2018)
- [Eigendecomposition-free Training of Deep Networks with Zero Eigenvalue-based Losses](https://mlsanity.com/paper/1803.08071/) (2018)

## Built on by (most cited first, 19 of 19)

- [Shape-Constraint Recurrent Flow for 6D Object Pose Estimation](https://mlsanity.com/paper/2306.13266/) (2023)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Self-Consistency Training for Density-Functional-Theory Hamiltonian Prediction](https://mlsanity.com/paper/2403.09560/) (2024)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Grouping-matrix based Graph Pooling with Adaptive Number of Clusters](https://mlsanity.com/paper/2209.02939/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Spectral Norm of Convolutional Layers with Circular and Zero Paddings](https://mlsanity.com/paper/2402.00240/) (2024)
- [Learning Unorthogonalized Matrices for Rotation Estimation](https://mlsanity.com/paper/2312.00462/) (2023)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [Deep unrolling Shrinkage Network for Dynamic MR imaging](https://mlsanity.com/paper/2307.09818/) (2023)
- [Iterative Low-rank Network for Hyperspectral Image Denoising](https://mlsanity.com/paper/2509.00356/) (2025)
- [JotlasNet: Joint Tensor Low-Rank and Attention-based Sparse Unrolling Network for Accelerating Dynamic MRI](https://mlsanity.com/paper/2502.11749/) (2025)
- [RadarLCD: Learnable Radar-based Loop Closure Detection Pipeline](https://mlsanity.com/paper/2309.07094/) (2023)
- [Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations](https://mlsanity.com/paper/2604.05414/) (2026)
- [Computation-Aware Kalman Filtering with Model Selection for Neural Dynamics](https://mlsanity.com/paper/2606.01468/) (2026)
- [Equivariant Neural Belief Propagation](https://mlsanity.com/paper/2606.06344/) (2026)
- [Fast Differentiable SVD on GPU via Polar Decomposition](https://mlsanity.com/paper/2609.32505/) (2026)

## Abstract (arXiv)

Eigendecomposition of symmetric matrices is at the heart of many computer vision algorithms. However, the derivatives of the eigenvectors tend to be numerically unstable, whether using the SVD to compute them analytically or using the Power Iteration (PI) method to approximate them. This instability arises in the presence of eigenvalues that are close to each other. This makes integrating eigendecomposition into deep networks difficult and often results in poor convergence, particularly when dealing with large matrices. While this can be mitigated by partitioning the data into small arbitrary groups, doing so has no theoretical basis and makes it impossible to exploit the full power of eigendecomposition. In previous work, we mitigated this using SVD during the forward pass and PI to compute the gradients during the backward pass. However, the iterative deflation procedure required to compute multiple eigenvectors using PI tends to accumulate errors and yield inaccurate gradients. Here, we show that the Taylor expansion of the SVD gradient is theoretically equivalent to the gradient obtained using PI without relying in practice on an iterative process and thus yields more accurate gradients. We demonstrate the benefits of this increased accuracy for image classification and style transfer.