# Structure-Preserving Transformers for Sequences of SPD Matrices

arXiv: https://arxiv.org/abs/2309.07579 · HTML: https://mlsanity.com/paper/2309.07579/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [A Riemannian Network for SPD Matrix Learning](https://mlsanity.com/paper/1608.04233/) (2016)
- [DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG](https://mlsanity.com/paper/1703.04046/) (2017)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification](https://mlsanity.com/paper/2105.11043/) (2021)
- [MAtt: A Manifold Attention Network for EEG Decoding](https://mlsanity.com/paper/2210.01986/) (2022)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [RobustSleepNet: Transfer learning for automated sleep staging at scale](https://mlsanity.com/paper/2101.02452/) (2021)
- [From Manifold to Manifold: Geometry-Aware Dimensionality Reduction for SPD Matrices](https://mlsanity.com/paper/1407.1120/) (2014)
- [L-SeqSleepNet: Whole-cycle Long Sequence Modelling for Automatic Sleep Staging](https://mlsanity.com/paper/2301.03441/) (2023)
- [Multi-manifold Attention for Vision Transformers](https://mlsanity.com/paper/2207.08569/) (2022)

## Built on by (most cited first, 4 of 4)

- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)
- [Subgroups of U(d) Induce Natural RNN and Transformer Architectures](https://mlsanity.com/paper/2602.18417/) (2026)
- [Automatic Classification of Sleep Stages from EEG Signals Using Riemannian Metrics and Transformer Networks](https://mlsanity.com/paper/2410.19819/) (2024)
- [A Unified SPD Token Transformer Framework for EEG Classification: Systematic Comparison of Geometric Embeddings](https://mlsanity.com/paper/2601.21521/) (2026)

## Abstract (arXiv)

In recent years, Transformer-based auto-attention mechanisms have been successfully applied to the analysis of a variety of context-reliant data types, from texts to images and beyond, including data from non-Euclidean geometries. In this paper, we present such a mechanism, designed to classify sequences of Symmetric Positive Definite matrices while preserving their Riemannian geometry throughout the analysis. We apply our method to automatic sleep staging on timeseries of EEG-derived covariance matrices from a standard dataset, obtaining high levels of stage-wise performance.