mlsanityData to 7 Oct 2026

Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentation

arXiv 2508.04513 ↗First posted 6 Aug 2025cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Current state-of-the-art methods for skeleton-based temporal action segmentation are predominantly supervised and require annotated data, which is expensive to collect. In contrast, existing unsupervised temporal action segmentation methods have focused primarily on video data, while skeleton sequences remain underexplored, despite their relevance to real-world applications, robustness, and privacy-preserving nature. In this paper, we propose a novel approach for unsupervised skeleton-based temporal action segmentation. Our method utilizes a sequence-to-sequence temporal autoencoder that keeps the information of the different joints disentangled in the embedding space. Latent skeleton sequences are then divided into non-overlapping patches and quantized to obtain distinctive skeleton motion words, driving the discovery of semantically meaningful action clusters. We thoroughly evaluate the proposed approach on three widely used skeleton-based datasets, namely HuGaDB, LARa, and BABEL. The results demonstrate that our model outperforms the current state-of-the-art unsupervised temporal action segmentation methods. Code is available at https://github.com/bachlab/SMQ .

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 50 of its 73 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

50 of 73 references.

References · 50

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  10. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  11. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  12. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
38 more
  1. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  2. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  3. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  4. 2021ASFormer: Transformer for Action Segmentationcited by 166
  5. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 105
  6. 2017PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understandingcited by 101
  7. 2019Efficient Parameter-free Clustering Using First Neighbor Relationscited by 96
  8. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
  9. 2019PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognitioncited by 90
  10. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 89
  11. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  12. 2016Segmental Spatiotemporal CNNs for Fine-grained Action Segmentationcited by 78
  13. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  14. 2023Diffusion Action Segmentationcited by 74
  15. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  16. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  17. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  18. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  19. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  20. 2023Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognitioncited by 41
  21. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  22. 2019A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentationcited by 36
  23. 2022Contrastive Positive Mining for Unsupervised 3D Action Representation Learningcited by 34
  24. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  25. 2022Skeleton-Based Action Segmentation with Multi-Stage Spatial-Temporal Graph Convolutional Neural Networkscited by 20
  26. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  27. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  28. 2017HuGaDB: Human Gait Database for Activity Recognition from Wearable Inertial Sensor Networkscited by 16
  29. 2021Unsupervised Motion Representation Learning with Capsule Autoencoderscited by 16
  30. 2023LAC: Latent Action Composition for Skeleton-based Action Segmentationcited by 15
  31. 2022Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariancecited by 10
  32. 2023Leveraging triplet loss for unsupervised action segmentationcited by 10
  33. 2015Efficient Unsupervised Temporal Segmentation of Motion Datacited by 7
  34. 2022An Efficient Framework for Few-shot Skeleton-based Temporal Action Segmentationcited by 7
  35. 2023A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentationcited by 7
  36. 2019Deep-Aligned Convolutional Neural Network for Skeleton-based Action Recognition and Segmentationcited by 3
  37. 2015Unsupervised Temporal Segmentation of Repetitive Human Actions Based on Kinematic Modeling and Frequency Analysiscited by 2
  38. 2023TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clusteringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detectioncited by 1
  2. 2025Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localizationcited by 0
  3. 2026Learning Quantised Structure-Preserving Motion Representations for Dance Fingerprintingcited by 0
  4. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0
  5. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 0
  6. 2026Improving Viewpoint-Invariance and Temporal Consistency for Action Detectioncited by 0