mlsanityData to 14 Oct 2026

Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment

arXiv 2305.19478 ↗First posted 31 May 2023cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation. Our code and dataset are available on our research website: https://retrocausal.ai/research/.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

47 of 70 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  8. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  9. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  10. 2014Deeply-Supervised Netscited by 367
  11. 2021ASFormer: Transformer for Action Segmentationcited by 168
  12. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
35 more
  1. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  2. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  3. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  4. 2019Oops! Predicting Unintentional Action in Videocited by 82
  5. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 76
  6. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  7. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  8. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  9. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  10. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  11. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  12. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  13. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  14. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 49
  15. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  16. 2021Learning by Aligning Videos in Timecited by 48
  17. 2016Weakly supervised learning of actions from transcriptscited by 46
  18. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  19. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  20. 2020SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentationcited by 39
  21. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  22. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  23. 2015Unsupervised Semantic Parsing of Video Collectionscited by 26
  24. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  25. 2022Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignmentscited by 23
  26. 2016Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsingcited by 19
  27. 2018Deep Supervision with Intermediate Conceptscited by 19
  28. 2020MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 19
  29. 2022Timestamp-Supervised Action Segmentation with Graph Convolutional Networkscited by 18
  30. 2020Towards Anomaly Detection in Dashcam Videoscited by 16
  31. 2021SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentationcited by 15
  32. 2017TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentationcited by 14
  33. 2019Spatio-Temporal Pyramid Graph Convolutions for Human Action Recognition and Postural Assessmentcited by 9
  34. 2022Robust Action Segmentation from Timestamp Supervisioncited by 6
  35. 2015Unsupervised Temporal Segmentation of Repetitive Human Actions Based on Kinematic Modeling and Frequency Analysiscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 6; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 38
  2. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 19
  3. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  4. 2023Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusioncited by 6
  5. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  6. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  7. 2025Procedure Learning via Regularized Gromov-Wasserstein Optimal Transportcited by 4
  8. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  9. 2025Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actionscited by 1
  10. 2025Biomechanical-phase based Temporal Segmentation in Sports Videos: a Demonstration on Javelin-Throwcited by 1
  11. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 1
  12. 2026Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approachcited by 1
5 more
  1. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0
  2. 2026From Division to Decision: Leveraging Temporal Cell-Stage Segmentation for Embryo Transferability Predictioncited by 0
  3. 2026D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 0
  4. 2026FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentationcited by 0
  5. 2026Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videoscited by 0