# Deep manifold-to-manifold transforming network for action recognition

arXiv: https://arxiv.org/abs/1705.10732 · HTML: https://mlsanity.com/paper/1705.10732/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2017.

## Built on by (most cited first, 9 of 9)

- [Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices](https://mlsanity.com/paper/2110.13475/) (2021)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis](https://mlsanity.com/paper/2203.01631/) (2022)
- [Mano: Restriking Manifold Optimization for LLM Training](https://mlsanity.com/paper/2601.23000/) (2026)
- [Modeling Graphs Beyond Hyperbolic: Graph Neural Networks in Symmetric Positive Definite Matrices](https://mlsanity.com/paper/2306.14064/) (2023)
- [Learning Chebyshev Basis in Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.05482/) (2021)
- [Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition](https://mlsanity.com/paper/2105.05708/) (2021)
- [Multiview Manifold Evidential Fusion for PolSAR Image Classification](https://mlsanity.com/paper/2510.11171/) (2025)
- [Geometric Feature Learning for Functional Data Valued on the Symmetric Positive Definite Manifold](https://mlsanity.com/paper/2609.30487/) (2026)

## Abstract (arXiv)

Symmetric positive definite (SPD) matrices (e.g., covariances, graph Laplacians, etc.) are widely used to model the relationship of spatial or temporal domain. Nevertheless, SPD matrices are theoretically embedded on Riemannian manifolds. In this paper, we propose an end-to-end deep manifold-to-manifold transforming network (DMT-Net) which can make SPD matrices flow from one Riemannian manifold to another more discriminative one. To learn discriminative SPD features characterizing both spatial and temporal dependencies, we specifically develop three novel layers on manifolds: (i) the local SPD convolutional layer, (ii) the non-linear SPD activation layer, and (iii) the Riemannian-preserved recursive layer. The SPD property is preserved through all layers without any requirement of singular value decomposition (SVD), which is often used in the existing methods with expensive computation cost. Furthermore, a diagonalizing SPD layer is designed to efficiently calculate the final metric for the classification task. To evaluate our proposed method, we conduct extensive experiments on the task of action recognition, where input signals are popularly modeled as SPD matrices. The experimental results demonstrate that our DMT-Net is much more competitive over state-of-the-art.