# Deep Learning on Lie Groups for Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/1612.05877 · HTML: https://mlsanity.com/paper/1612.05877/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2016.

## Built on by (most cited first, 30 of 42)

- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Universal Approximation Theorems for Differentiable Geometric Deep Learning](https://mlsanity.com/paper/2101.05390/) (2021)
- [AnyTop: Character Animation Diffusion with Any Topology](https://mlsanity.com/paper/2502.17327/) (2025)
- [GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition](https://mlsanity.com/paper/2111.13089/) (2021)
- [PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation](https://mlsanity.com/paper/2209.08194/) (2022)
- [Tensor-CSPNet: A Novel Geometric Deep Learning Framework for Motor Imagery Classification](https://mlsanity.com/paper/2202.02472/) (2022)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Group Equivariant Conditional Neural Processes](https://mlsanity.com/paper/2102.08759/) (2021)
- [Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble](https://mlsanity.com/paper/2110.06161/) (2021)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [Panoramic Human Activity Recognition](https://mlsanity.com/paper/2203.03806/) (2022)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [A Lie Group Approach to Riemannian Batch Normalization](https://mlsanity.com/paper/2403.11261/) (2024)
- [RMLR: Extending Multinomial Logistic Regression into General Geometries](https://mlsanity.com/paper/2409.19433/) (2024)
- [Deep Optimal Transport for Domain Adaptation on SPD Manifolds](https://mlsanity.com/paper/2201.05745/) (2022)
- [CameraPose: Weakly-Supervised Monocular 3D Human Pose Estimation by Leveraging In-the-wild 2D Annotations](https://mlsanity.com/paper/2301.02979/) (2023)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Neural Networks on Symmetric Spaces of Noncompact Type](https://mlsanity.com/paper/2601.01097/) (2026)
- [tensorflow-riemopt: A Library for Optimization on Riemannian Manifolds](https://mlsanity.com/paper/2105.13921/) (2021)
- [Intrinsic and extrinsic deep learning on manifolds](https://mlsanity.com/paper/2302.08606/) (2023)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [Fast and Stable Riemannian Metrics on SPD Manifolds via Cholesky Product Geometry](https://mlsanity.com/paper/2407.02607/) (2024)
- [Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition](https://mlsanity.com/paper/2503.20436/) (2025)
- [GEM: Group Enhanced Model for Learning Dynamical Control Systems](https://mlsanity.com/paper/2104.02844/) (2021)
- [CGA-PoseNet: Camera Pose Regression via a 1D-Up Approach to Conformal Geometric Algebra](https://mlsanity.com/paper/2302.05211/) (2023)
- [Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size](https://mlsanity.com/paper/2501.18164/) (2025)
- [Rieoptax: Riemannian Optimization in JAX](https://mlsanity.com/paper/2210.04840/) (2022)
- [Siegel Neural Networks](https://mlsanity.com/paper/2511.09577/) (2025)
- [Learning Chebyshev Basis in Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.05482/) (2021)
- [Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition](https://mlsanity.com/paper/2105.05708/) (2021)

## Abstract (arXiv)

In recent years, skeleton-based action recognition has become a popular 3D classification problem. State-of-the-art methods typically first represent each motion sequence as a high-dimensional trajectory on a Lie group with an additional dynamic time warping, and then shallowly learn favorable Lie group features. In this paper we incorporate the Lie group structure into a deep network architecture to learn more appropriate Lie group features for 3D action recognition. Within the network structure, we design rotation mapping layers to transform the input Lie group features into desirable ones, which are aligned better in the temporal domain. To reduce the high feature dimensionality, the architecture is equipped with rotation pooling layers for the elements on the Lie group. Furthermore, we propose a logarithm mapping layer to map the resulting manifold data into a tangent space that facilitates the application of regular output layers for the final classification. Evaluations of the proposed network for standard 3D human action recognition datasets clearly demonstrate its superiority over existing shallow Lie group feature learning methods as well as most conventional deep learning methods.