mlsanityData to 7 Oct 2026

Hierarchical Vector Quantization for Unsupervised Action Segmentation

arXiv 2412.17640 ↗First posted 23 Dec 2024cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

In this work, we address unsupervised temporal action segmentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine representation learning and clustering in a single step for this task, they do not cope with large variations within temporal segments of the same class. To address this limitation, we propose a novel method, termed Hierarchical Vector Quantization (HVQ), that consists of two subsequent vector quantization modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a cluster. We demonstrate that our approach captures the distribution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three public datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

38 of 52 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  10. 2020Jukebox: A Generative Model for Musiccited by 535
  11. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  12. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
26 more
  1. 2021ASFormer: Transformer for Action Segmentationcited by 166
  2. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  3. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 86
  4. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  5. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  6. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  7. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  8. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  9. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  10. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  11. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  12. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  13. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  14. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  15. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  16. 2016Weakly supervised learning of actions from transcriptscited by 46
  17. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  18. 2020SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentationcited by 39
  19. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  20. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  21. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  22. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  23. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  24. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  25. 2021FIFA: Fast Inference Approximation for Action Segmentationcited by 16
  26. 2023TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clusteringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 12.

18 papers.

Sort:

By area:

  1. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  2. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  3. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  4. 2025Temporal Rate Reduction Clustering for Human Motion Segmentationcited by 1
  5. 2025Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actionscited by 1
  6. 2026Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentationcited by 1
  7. 2026Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approachcited by 1
  8. 2025Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activitiescited by 0
  9. 2026Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Roboticscited by 0
  10. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0
  11. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 0
  12. 2026Deep kernel video approximation for unsupervised action segmentationcited by 0
6 more
  1. 2026Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentationcited by 0
  2. 2026D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 0
  3. 2026GRACE: LLM-Grounded Semantic Metric Spaces for Scalable Mixed-Data Clusteringcited by 0
  4. 2026See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learningcited by 0
  5. 2026FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentationcited by 0
  6. 2026Rondo: Unsupervised Discovery of Recurring Temporal Structurecited by 0