# Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation

arXiv: https://arxiv.org/abs/2404.01518 · HTML: https://mlsanity.com/paper/2404.01518/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Optimal Transport for structured data with application on graphs](https://mlsanity.com/paper/1805.09114/) (2018)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Gromov-Wasserstein Learning for Graph Matching and Node Embedding](https://mlsanity.com/paper/1901.06003/) (2019)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [Fused Gromov-Wasserstein distance for structured objects: theoretical foundations and mathematical properties](https://mlsanity.com/paper/1811.02834/) (2018)
- [The Fast Bilateral Solver](https://mlsanity.com/paper/1511.03296/) (2015)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)
- [Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation](https://mlsanity.com/paper/2003.02824/) (2020)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation](https://mlsanity.com/paper/2009.04266/) (2020)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences](https://mlsanity.com/paper/2001.11122/) (2020)
- [Accurate Point Cloud Registration with Robust Optimal Transport](https://mlsanity.com/paper/2111.00648/) (2021)
- [Action Shuffle Alternating Learning for Unsupervised Action Segmentation](https://mlsanity.com/paper/2104.02116/) (2021)
- [Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities](https://mlsanity.com/paper/2105.00067/) (2021)
- [Aligning individual brains with Fused Unbalanced Gromov-Wasserstein](https://mlsanity.com/paper/2206.09398/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [Unbalanced Optimal Transport: A Unified Framework for Object Detection](https://mlsanity.com/paper/2307.02402/) (2023)
- [Efficient Approximation of Gromov-Wasserstein Distance Using Importance Sparsification](https://mlsanity.com/paper/2205.13573/) (2022)

## Built on by (most cited first, 30 of 38)

- [Hierarchical Vector Quantization for Unsupervised Action Segmentation](https://mlsanity.com/paper/2412.17640/) (2024)
- [Joint Self-Supervised Video Alignment and Action Segmentation](https://mlsanity.com/paper/2503.16832/) (2025)
- [Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport](https://mlsanity.com/paper/2503.15337/) (2025)
- [CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2507.03539/) (2025)
- [Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentation](https://mlsanity.com/paper/2508.04513/) (2025)
- [Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model](https://mlsanity.com/paper/2407.04489/) (2024)
- [Procedure Learning via Regularized Gromov-Wasserstein Optimal Transport](https://mlsanity.com/paper/2507.15540/) (2025)
- [2by2: Weakly-Supervised Learning for Global Action Segmentation](https://mlsanity.com/paper/2412.12829/) (2024)
- [SP2OT: Semantic-Regularized Progressive Partial Optimal Transport for Imbalanced Clustering](https://mlsanity.com/paper/2404.03446/) (2024)
- [Temporal Rate Reduction Clustering for Human Motion Segmentation](https://mlsanity.com/paper/2506.21249/) (2025)
- [Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actions](https://mlsanity.com/paper/2508.05529/) (2025)
- [REMAP: Regularized Matching and Partial Alignment of Video Embeddings](https://mlsanity.com/paper/2509.24382/) (2025)
- [Biomechanical-phase based Temporal Segmentation in Sports Videos: a Demonstration on Javelin-Throw](https://mlsanity.com/paper/2509.24606/) (2025)
- [Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation](https://mlsanity.com/paper/2602.21406/) (2026)
- [Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning](https://mlsanity.com/paper/2603.11460/) (2026)
- [Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild](https://mlsanity.com/paper/2603.11618/) (2026)
- [A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics](https://mlsanity.com/paper/2604.15215/) (2026)
- [Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approach](https://mlsanity.com/paper/2607.05271/) (2026)
- [Learning Satellite Pattern-of-Life Identification: A Diffusion-based Approach](https://mlsanity.com/paper/2412.10814/) (2024)
- [Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach](https://mlsanity.com/paper/2507.03458/) (2025)
- [Learning local and global prototypes with optimal transport for unsupervised anomaly detection and localization](https://mlsanity.com/paper/2508.12927/) (2025)
- [Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activities](https://mlsanity.com/paper/2511.14945/) (2025)
- [Unsupervised Hierarchical Skill Discovery](https://mlsanity.com/paper/2601.23156/) (2026)
- [Gromov Wasserstein Optimal Transport for Semantic Correspondences](https://mlsanity.com/paper/2602.03105/) (2026)
- [Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation](https://mlsanity.com/paper/2604.01859/) (2026)
- [Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization](https://mlsanity.com/paper/2604.15196/) (2026)
- [Deep kernel video approximation for unsupervised action segmentation](https://mlsanity.com/paper/2604.21572/) (2026)
- [Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentation](https://mlsanity.com/paper/2605.05753/) (2026)
- [Improving Viewpoint-Invariance and Temporal Consistency for Action Detection](https://mlsanity.com/paper/2605.22695/) (2026)
- [OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classification](https://mlsanity.com/paper/2606.19838/) (2026)

## Abstract (arXiv)

We propose a novel approach to the action segmentation task for long, untrimmed videos, based on solving an optimal transport problem. By encoding a temporal consistency prior into a Gromov-Wasserstein problem, we are able to decode a temporally consistent segmentation from a noisy affinity/matching cost matrix between video frames and action classes. Unlike previous approaches, our method does not require knowing the action order for a video to attain temporal consistency. Furthermore, our resulting (fused) Gromov-Wasserstein problem can be efficiently solved on GPUs using a few iterations of projected mirror descent. We demonstrate the effectiveness of our method in an unsupervised learning setting, where our method is used to generate pseudo-labels for self-training. We evaluate our segmentation approach and unsupervised learning pipeline on the Breakfast, 50-Salads, YouTube Instructions and Desktop Assembly datasets, yielding state-of-the-art results for the unsupervised video action segmentation task.