mlsanityData to 13 Oct 2026

CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation

arXiv 2507.03539 ↗First posted 4 Jul 2025cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent segmentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leveraging its encoder-decoder architecture, CLOT learns pseudo-labels alongside frame and segment embeddings by solving two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention between the learned frame and segment embeddings, by integrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cyclical learning for unsupervised action segmentation.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

39 of 53 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,106
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2020End-to-End Object Detection with Transformerscited by 7,396
  4. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  5. 2019Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptationcited by 235
  6. 2019Generalized Sliced Wasserstein Distancescited by 206
  7. 2021Anticipative Video Transformercited by 199
  8. 2021ASFormer: Transformer for Action Segmentationcited by 167
  9. 2019Max-Sliced Wasserstein Distance and its use for GANscited by 118
  10. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  11. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  12. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
27 more
  1. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  2. 2019Sliced Wasserstein Generative Modelscited by 61
  3. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  4. 2022Future Transformer for Long-term Action Anticipationcited by 58
  5. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  6. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  7. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  8. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  9. 2023Energy-Based Sliced Wasserstein Distancecited by 39
  10. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  11. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  12. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  13. 2019Fast Weakly Supervised Action Segmentation Using Mutual Consistencycited by 30
  14. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  15. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  16. 2021Learning Discriminative Prototypes with Dynamic Time Warpingcited by 23
  17. 2017Wasserstein Introspective Neural Networkscited by 18
  18. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  19. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  20. 2021Temporal Action Segmentation with High-level Complex Activity Labelscited by 16
  21. 2024Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignmentcited by 13
  22. 2023OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentationcited by 11
  23. 2023Leveraging triplet loss for unsupervised action segmentationcited by 10
  24. 2024ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysiscited by 7
  25. 2023ClusterFormer: Clustering As A Universal Visual Learnercited by 6
  26. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  27. 2024Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Cameracited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approachcited by 1
  2. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0
  3. 2026Deep kernel video approximation for unsupervised action segmentationcited by 0
  4. 2026Adaptive Latent Trajectory Anchoring for Action Segmentation Dataset Condensationcited by 0
  5. 2026D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 0
  6. 2026See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learningcited by 0