# An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

arXiv: https://arxiv.org/abs/2605.09231 · HTML: https://mlsanity.com/paper/2605.09231/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [geomstats: a Python Package for Riemannian Geometry in Machine Learning](https://mlsanity.com/paper/1805.08308/) (2018)
- [SkeletonGait: Gait Recognition Using Skeleton Maps](https://mlsanity.com/paper/2311.13444/) (2023)
- [Variational Autoencoders with Riemannian Brownian Motion Priors](https://mlsanity.com/paper/2002.05227/) (2020)
- [Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder](https://mlsanity.com/paper/2105.00026/) (2021)
- [Conditional Image Generation by Conditioning Variational Auto-Encoders](https://mlsanity.com/paper/2102.12037/) (2021)
- [A Geometric Perspective on Variational Autoencoders](https://mlsanity.com/paper/2209.07370/) (2022)
- [Learning Weighted Submanifolds with Variational Autoencoders and Riemannian Variational Autoencoders](https://mlsanity.com/paper/1911.08147/) (2019)
- [Adaptive Hyper-Graph Convolution Network for Skeleton-based Human Action Recognition with Virtual Connections](https://mlsanity.com/paper/2411.14796/) (2024)
- [VTAE: Variational Transformer Autoencoder with Manifolds Learning](https://mlsanity.com/paper/2304.00948/) (2023)
- [RDA-INR: Riemannian Diffeomorphic Autoencoding via Implicit Neural Representations](https://mlsanity.com/paper/2305.12854/) (2023)
- [Benchmarking Skeleton-based Motion Encoder Models for Clinical Applications: Estimating Parkinson's Disease Severity in Walking Sequences](https://mlsanity.com/paper/2405.17817/) (2024)
- [Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?](https://mlsanity.com/paper/2505.10679/) (2025)
- [ManifoldFormer: Geometric Deep Learning for Neural Dynamics on Riemannian Manifolds](https://mlsanity.com/paper/2511.16828/) (2025)
- [A-MAL: Automatic Movement Assessment Learning from Properly Performed Movements in 3D Skeleton Videos](https://mlsanity.com/paper/1907.10004/) (2019)

## Abstract (arXiv)

Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors-such as camera orientation, subject scale, viewpoint, and execution speed-rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall's shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches.