mlsanityData to 7 Oct 2026

See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning

arXiv 2608.29611 ↗First posted 30 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Unsupervised action segmentation aims to discover latent action categories and their temporal organization without action annotations. Optimal transport-based methods provide structured frame-to-action assignments, however, their pseudo-label quality is fundamentally conditioned on the representation space used to construct the transport cost. We argue that reliable OT pseudo-labeling requires a representation geometry that is simultaneously sensitive to discriminative action changes and coherent along local temporal progressions. Based on this insight, we propose SpecT-OT, a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept. SpecT-OT introduces a Spectral Reparameterization Projector (SRP), which parameterizes projector weights with fixed Fourier bases and learnable coefficients to improve the modeling of rapidly varying discriminative features, and Temporal Affinity Regularization (TAR), which imposes distance-aware, label-free constraints on pairwise frame affinities to stabilize local temporal structure. The two components jointly produce more discriminative and temporally stable transport costs, yielding more reliable pseudo-labels for iterative representation learning. Experiments on four benchmarks demonstrate strong performance compared with state-of-the-art methods. SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly, respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

30 of 38 references.

References · 30

  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  4. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  5. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  6. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  7. 2021ASFormer: Transformer for Action Segmentationcited by 166
  8. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  9. 2017CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videoscited by 112
  10. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 86
  11. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  12. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
18 more
  1. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  2. 2017DiracNets: Training Very Deep Neural Networks Without Skip-Connectionscited by 47
  3. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  4. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  5. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  6. 2024USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020scited by 37
  7. 2024Improved Implicit Neural Representation with Fourier Reparameterized Trainingcited by 36
  8. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  9. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  10. 2024Object Segmentation-Assisted Inter Prediction for Versatile Video Codingcited by 22
  11. 2021Domain Adaptive Video Segmentation via Temporal Consistency Regularizationcited by 21
  12. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  13. 2024Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignmentcited by 13
  14. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  15. 2022Deep Learning Methods for Calibrated Photometric Stereo and Beyondcited by 10
  16. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  17. 2026Just Noticeable Difference Modeling for Deep Visual Featurescited by 3
  18. 2026Learning Probabilistic Embeddings for Unsupervised Action Segmentationcited by 1

Built on by

Not cited yet.