# Riemannian Self-Attention Mechanism for SPD Networks

arXiv: https://arxiv.org/abs/2311.16738 · HTML: https://mlsanity.com/paper/2311.16738/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [A Riemannian Network for SPD Matrix Learning](https://mlsanity.com/paper/1608.04233/) (2016)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles](https://mlsanity.com/paper/2104.00946/) (2021)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Riemannian batch normalization for SPD neural networks](https://mlsanity.com/paper/1909.02414/) (2019)
- [Building Deep Networks on Grassmann Manifolds](https://mlsanity.com/paper/1611.05742/) (2016)
- [A neural network based on SPD manifold learning for skeleton-based hand gesture recognition](https://mlsanity.com/paper/1904.12970/) (2019)
- [Dimensionality Reduction on SPD Manifolds: The Emergence of Geometry-Aware Methods](https://mlsanity.com/paper/1605.06182/) (2016)
- [Riemannian Local Mechanism for SPD Neural Networks](https://mlsanity.com/paper/2201.10145/) (2022)
- [Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping](https://mlsanity.com/paper/1906.05947/) (2019)
- [Transition Forests: Learning Discriminative Temporal Transitions for Action Recognition and Detection](https://mlsanity.com/paper/1607.02737/) (2016)
- [O-ViT: Orthogonal Vision Transformer](https://mlsanity.com/paper/2201.12133/) (2022)
- [Sparse Coding and Dictionary Learning for Symmetric Positive Definite Matrices: A Kernel Approach](https://mlsanity.com/paper/1304.4344/) (2013)
- [Analyzing Dynamical Brain Functional Connectivity As Trajectories on Space of Covariance Matrices](https://mlsanity.com/paper/1904.05449/) (2019)
- [Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning](https://mlsanity.com/paper/1908.01950/) (2019)
- [Spatio-Temporal Covariance Descriptors for Action and Gesture Recognition](https://mlsanity.com/paper/1303.6021/) (2013)

## Built on by (most cited first, 4 of 4)

- [Subgroups of U(d) Induce Natural RNN and Transformer Architectures](https://mlsanity.com/paper/2602.18417/) (2026)
- [CliffSearch: Structured Agentic Co-Evolution over Theory and Code for Scientific Algorithm Discovery](https://mlsanity.com/paper/2604.01210/) (2026)
- [Escaping the Capacity Ceiling: Routing on the Stiefel Manifold for Bilinear SPD Layers](https://mlsanity.com/paper/2605.31043/) (2026)
- [SPD-MetaFormer is what you need for small-data brain decoding](https://mlsanity.com/paper/2610.10952/) (2026)

## Abstract (arXiv)

Symmetric positive definite (SPD) matrix has been demonstrated to be an effective feature descriptor in many scientific areas, as it can encode spatiotemporal statistics of the data adequately on a curved Riemannian manifold, i.e., SPD manifold. Although there are many different ways to design network architectures for SPD matrix nonlinear learning, very few solutions explicitly mine the geometrical dependencies of features at different layers. Motivated by the great success of self-attention mechanism in capturing long-range relationships, an SPD manifold self-attention mechanism (SMSA) is proposed in this paper using some manifold-valued geometric operations, mainly the Riemannian metric, Riemannian mean, and Riemannian optimization. Then, an SMSA-based geometric learning module (SMSA-GLM) is designed for the sake of improving the discrimination of the generated deep structured representations. Extensive experimental results achieved on three benchmarking datasets show that our modification against the baseline network further alleviates the information degradation problem and leads to improved accuracy.