mlsanityData to 7 Oct 2026

Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation

arXiv 2103.11264 ↗First posted 20 Mar 2021cs.CV cs.AI cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Action segmentation refers to inferring boundaries of semantically consistent visual concepts in videos and is an important requirement for many video understanding tasks. For this and other video understanding tasks, supervised approaches have achieved encouraging performance but require a high volume of detailed frame-level annotations. We present a fully automatic and unsupervised approach for segmenting actions in a video that does not require any training. Our proposal is an effective temporally-weighted hierarchical clustering algorithm that can group semantically consistent frames of the video. Our main finding is that representing a video with a 1-nearest neighbor graph by taking into account the time progression is sufficient to form semantically and temporally consistent clusters of frames where each cluster may represent some action in the video. Additionally, we establish strong unsupervised baselines for action segmentation and show significant performance improvements over published unsupervised methods on five challenging action segmentation datasets. Our code is available at https://github.com/ssarfraz/FINCH-Clustering/tree/master/TW-FINCH

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

27 of 42 references.

References · 27

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  4. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  5. 2019Efficient Parameter-free Clustering Using First Neighbor Relationscited by 96
  6. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  7. 2016Segmental Spatiotemporal CNNs for Fine-grained Action Segmentationcited by 78
  8. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  9. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  10. 2020Action Segmentation with Joint Self-Supervised Temporal Domain Adaptationcited by 74
  11. 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
  12. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
15 more
  1. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  2. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  3. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  4. 2016Going Deeper into Action Recognition: A Surveycited by 55
  5. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  6. 2014Weakly Supervised Action Labeling in Videos Under Ordering Constraintscited by 48
  7. 2016Weakly supervised learning of actions from transcriptscited by 46
  8. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  9. 2015Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Datacited by 40
  10. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  11. 2020SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentationcited by 39
  12. 2019A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentationcited by 36
  13. 2019Fast Weakly Supervised Action Segmentation Using Mutual Consistencycited by 30
  14. 2020MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 18
  15. 2017TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 11; 2024: 8; 2025: 15; 2026: 11.

58 papers.

Sort:

By area:

  1. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  2. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  3. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  4. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  5. 2022C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentationcited by 23
  6. 2021Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentationcited by 19
  7. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  8. 2022Leveraging Action Affinity and Continuity for Semi-supervised Temporal Action Segmentationcited by 17
  9. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  10. 2021Temporal Action Segmentation with High-level Complex Activity Labelscited by 16
  11. 2021SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentationcited by 15
  12. 2021UnweaveNet: Unweaving Activity Storiescited by 14
38 more
  1. 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
  2. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  3. 2023OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentationcited by 11
  4. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  5. 2023Leveraging triplet loss for unsupervised action segmentationcited by 10
  6. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  7. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  8. 2025HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videoscited by 7
  9. 2025TAG: A Simple Yet Effective Temporal-Aware Approach for Zero-Shot Video Temporal Groundingcited by 7
  10. 2022MHMS: Multimodal Hierarchical Multimedia Summarizationcited by 6
  11. 2022LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videoscited by 6
  12. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  13. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  14. 2023Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosiscited by 5
  15. 2024Coherent Temporal Synthesis for Incremental Action Segmentationcited by 5
  16. 2024Open-Vocabulary Action Localization with Iterative Visual Promptingcited by 5
  17. 2023Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modelingcited by 4
  18. 2023Collaborative Weakly Supervised Video Correlation Learning for Procedure-Aware Instructional Video Analysiscited by 4
  19. 2025Condensing Action Segmentation Datasets via Generative Network Inversioncited by 4
  20. 2022Unsupervised Audio-Visual Lecture Segmentationcited by 3
  21. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  22. 2023TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clusteringcited by 2
  23. 2025Beyond Label Semantics: Language-Guided Action Anatomy for Few-shot Action Recognitioncited by 2
  24. 2026ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QAcited by 2
  25. 2022Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptationscited by 1
  26. 2023Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domainscited by 1
  27. 2023SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentationcited by 1
  28. 2024Multimodal Fusion and Coherence Modeling for Video Topic Segmentationcited by 1
  29. 2025Improving action segmentation via explicit similarity measurementcited by 1
  30. 2025Temporal Rate Reduction Clustering for Human Motion Segmentationcited by 1
  31. 2025Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actionscited by 1
  32. 2026Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentationcited by 1
  33. 2026Hierarchical Action Learning for Weakly-Supervised Action Segmentationcited by 1
  34. 2026Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approachcited by 1
  35. 2023Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Featurescited by 0
  36. 2025Real-Time Assessment of Bystander Situation Awareness in Drone-Assisted First Aidcited by 0
  37. 2025Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activitiescited by 0
  38. 2025Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learningcited by 0

50 most cited of 58 All 58 (JSON)