# Training Deep Networks with Structured Layers by Matrix Backpropagation

arXiv: https://arxiv.org/abs/1509.07838 · HTML: https://mlsanity.com/paper/1509.07838/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2015.

## Built on by (most cited first, 30 of 30)

- [Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?](https://mlsanity.com/paper/2105.02498/) (2021)
- [SHARP Challenge 2023: Solving CAD History and pArameters Recovery from Point clouds and 3D scans. Overview, Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2308.15966/) (2023)
- [Deep Graph Matching under Quadratic Constraint](https://mlsanity.com/paper/2103.06643/) (2021)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [SepicNet: Sharp Edges Recovery by Parametric Inference of Curves in 3D Shapes](https://mlsanity.com/paper/2304.06531/) (2023)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [RMLR: Extending Multinomial Logistic Regression into General Geometries](https://mlsanity.com/paper/2409.19433/) (2024)
- [Learning to Normalize on the SPD Manifold under Bures-Wasserstein Geometry](https://mlsanity.com/paper/2504.00660/) (2025)
- [SoT: Delving Deeper into Classification Head for Transformer](https://mlsanity.com/paper/2104.10935/) (2021)
- [Grouping-matrix based Graph Pooling with Adaptive Number of Clusters](https://mlsanity.com/paper/2209.02939/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problems](https://mlsanity.com/paper/2412.11165/) (2024)
- [Differentiable Deconvolution for Improved Stroke Perfusion Analysis](https://mlsanity.com/paper/2103.17111/) (2021)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Grassmannian learning mutual subspace method for image set recognition](https://mlsanity.com/paper/2111.04352/) (2021)
- [Classification of Buried Objects from Ground Penetrating Radar Images by using Second Order Deep Learning Models](https://mlsanity.com/paper/2410.07117/) (2024)
- [Adversarial Representation Learning With Closed-Form Solvers](https://mlsanity.com/paper/2109.05535/) (2021)
- [GrFormer: A Novel Transformer on Grassmann Manifold for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2506.14384/) (2025)
- [DreamNet: A Deep Riemannian Network based on SPD Manifold Learning for Visual Classification](https://mlsanity.com/paper/2206.07967/) (2022)
- [VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models](https://mlsanity.com/paper/2503.19530/) (2025)
- [SVMax: A Feature Embedding Regularizer](https://mlsanity.com/paper/2103.02770/) (2021)
- [Grassmann Iterative Linear Discriminant Analysis with Proxy Matrix Optimization](https://mlsanity.com/paper/2104.08112/) (2021)
- [Learning Topology-Driven Multi-Subspace Fusion for Grassmannian Deep Network](https://mlsanity.com/paper/2511.08628/) (2025)
- [SOLAR: SVD-Optimized Lifelong Attention for Recommendation](https://mlsanity.com/paper/2603.02561/) (2026)
- [Voronoi-based Second-order Descriptor with Whitened Metric in LiDAR Place Recognition](https://mlsanity.com/paper/2603.14974/) (2026)
- [PIEDet: Prototype-Driven Intrinsically Explainable Object Detection](https://mlsanity.com/paper/2604.13981/) (2026)
- [Riemannian Networks over Full-Rank Correlation Matrices](https://mlsanity.com/paper/2605.19073/) (2026)
- [FlexPooling with Simple Auxiliary Classifiers in Deep Networks](https://mlsanity.com/paper/2606.14926/) (2026)
- [Riemannian Deep Learning: Modules, Networks, and Geometries](https://mlsanity.com/paper/2607.19305/) (2026)

## Abstract (arXiv)

Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fields, and from the simplicity and scalability of the gradient-descent training procedure based on backpropagation. An open problem is the inclusion of layers that perform global, structured matrix computations like segmentation (e.g. normalized cuts) or higher-order pooling (e.g. log-tangent space metrics defined over the manifold of symmetric positive definite matrices) while preserving the validity and efficiency of an end-to-end deep training framework. In this paper we propose a sound mathematical apparatus to formally integrate global structured computation into deep computation architectures. At the heart of our methodology is the development of the theory and practice of backpropagation that generalizes to the calculus of adjoint matrix variations. The proposed matrix backpropagation methodology applies broadly to a variety of problems in machine learning or computational perception. Here we illustrate it by performing visual segmentation experiments using the BSDS and MSCOCO benchmarks, where we show that deep networks relying on second-order pooling and normalized cuts layers, trained end-to-end using matrix backpropagation, outperform counterparts that do not take advantage of such global layers.