mlsanityData to 7 Oct 2026

Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation

arXiv 2404.01518 ↗First posted 1 Apr 2024cs.CV cs.LG eess.IVcited by 37

Not on the map. Why?

Abstract (arXiv)

We propose a novel approach to the action segmentation task for long, untrimmed videos, based on solving an optimal transport problem. By encoding a temporal consistency prior into a Gromov-Wasserstein problem, we are able to decode a temporally consistent segmentation from a noisy affinity/matching cost matrix between video frames and action classes. Unlike previous approaches, our method does not require knowing the action order for a video to attain temporal consistency. Furthermore, our resulting (fused) Gromov-Wasserstein problem can be efficiently solved on GPUs using a few iterations of projected mirror descent. We demonstrate the effectiveness of our method in an unsupervised learning setting, where our method is used to generate pseudo-labels for self-training. We evaluate our segmentation approach and unsupervised learning pipeline on the Breakfast, 50-Salads, YouTube Instructions and Desktop Assembly datasets, yielding state-of-the-art results for the unsupervised video action segmentation task.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

29 of 44 references.

References · 29

  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  4. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  5. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  6. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  7. 2018Optimal Transport for structured data with application on graphscited by 170
  8. 2021ASFormer: Transformer for Action Segmentationcited by 166
  9. 2019Gromov-Wasserstein Learning for Graph Matching and Node Embeddingcited by 125
  10. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  11. 2018Fused Gromov-Wasserstein distance for structured objects: theoretical foundations and mathematical propertiescited by 93
  12. 2015The Fast Bilateral Solvercited by 86
17 more
  1. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 86
  2. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  3. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  4. 2020Action Segmentation with Joint Self-Supervised Temporal Domain Adaptationcited by 74
  5. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  6. 2020The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxationcited by 55
  7. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  8. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  9. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  10. 2021Accurate Point Cloud Registration with Robust Optimal Transportcited by 38
  11. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  12. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  13. 2022Aligning individual brains with Fused Unbalanced Gromov-Wassersteincited by 23
  14. 2021Domain Adaptive Video Segmentation via Temporal Consistency Regularizationcited by 21
  15. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  16. 2023Unbalanced Optimal Transport: A Unified Framework for Object Detectioncited by 14
  17. 2022Efficient Approximation of Gromov-Wasserstein Distance Using Importance Sparsificationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 20.

37 papers.

Sort:

By area:

  1. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  2. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  3. 2025Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transportcited by 6
  4. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  5. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  6. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Modelcited by 4
  7. 2025Procedure Learning via Regularized Gromov-Wasserstein Optimal Transportcited by 4
  8. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  9. 2024SP2OT: Semantic-Regularized Progressive Partial Optimal Transport for Imbalanced Clusteringcited by 2
  10. 2025Temporal Rate Reduction Clustering for Human Motion Segmentationcited by 1
  11. 2025Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actionscited by 1
  12. 2025REMAP: Regularized Matching and Partial Alignment of Video Embeddingscited by 1
25 more
  1. 2026Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentationcited by 1
  2. 2026Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioningcited by 1
  3. 2026Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approachcited by 1
  4. 2024Learning Satellite Pattern-of-Life Identification: A Diffusion-based Approachcited by 0
  5. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approachcited by 0
  6. 2025Learning local and global prototypes with optimal transport for unsupervised anomaly detection and localizationcited by 0
  7. 2025Biomechanical-phase based Temporal Segmentation in Sports Videos: a Demonstration on Javelin-Throwcited by 0
  8. 2025Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activitiescited by 0
  9. 2026Unsupervised Hierarchical Skill Discoverycited by 0
  10. 2026Gromov Wasserstein Optimal Transport for Semantic Correspondencescited by 0
  11. 2026Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wildcited by 0
  12. 2026Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentationcited by 0
  13. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0
  14. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 0
  15. 2026Deep kernel video approximation for unsupervised action segmentationcited by 0
  16. 2026Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentationcited by 0
  17. 2026Improving Viewpoint-Invariance and Temporal Consistency for Action Detectioncited by 0
  18. 2026OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classificationcited by 0
  19. 2026D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 0
  20. 2026ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videoscited by 0
  21. 2026See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learningcited by 0
  22. 2026FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentationcited by 0
  23. 2026CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generationcited by 0
  24. 2026Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videoscited by 0
  25. 2026Rondo: Unsupervised Discovery of Recurring Temporal Structurecited by 0