mlsanityData to 7 Oct 2026

OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation

arXiv 2309.06276 ↗First posted 12 Sep 2023cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Temporal action segmentation is typically achieved by discovering the dramatic variances in global visual descriptors. In this paper, we explore the merits of local features by proposing the unsupervised framework of Object-centric Temporal Action Segmentation (OTAS). Broadly speaking, OTAS consists of self-supervised global and local feature extraction modules as well as a boundary selection module that fuses the features and detects salient boundaries for action segmentation. As a second contribution, we discuss the pros and cons of existing frame-level and boundary-level evaluation metrics. Through extensive experiments, we find OTAS is superior to the previous state-of-the-art method by 41% on average in terms of our recommended F1 score. Surprisingly, OTAS even outperforms the ground-truth human annotations in the user study. Moreover, OTAS is efficient enough to allow real-time inference.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Graph Attention Networkscited by 8,854
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2021How Attentive are Graph Attention Networks?cited by 818
  9. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  11. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  12. 2018Visual Semantic Navigation using Scene Priorscited by 127
28 more
  1. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  2. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  3. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  4. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  5. 2020Action Segmentation with Joint Self-Supervised Temporal Domain Adaptationcited by 74
  6. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  7. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  8. 2021Generic Event Boundary Detection: A Benchmark for Event Segmentationcited by 64
  9. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  10. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  11. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  12. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  13. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  14. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  15. 2016Weakly supervised learning of actions from transcriptscited by 46
  16. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  17. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  18. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  19. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  20. 2015Unsupervised Semantic Parsing of Video Collectionscited by 26
  21. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  22. 2015Weakly-Supervised Alignment of Video With Textcited by 21
  23. 2020Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluationcited by 20
  24. 2021UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detectioncited by 20
  25. 2021Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detectioncited by 19
  26. 2021Temporal Action Segmentation with High-level Complex Activity Labelscited by 16
  27. 2021SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentationcited by 15
  28. 2021CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 5.

11 papers.

  1. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  2. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  3. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  4. 2025Improving action segmentation via explicit similarity measurementcited by 1
  5. 2025Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activitiescited by 0
  6. 2025From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-training in Industrial Settingscited by 0
  7. 2026ATLAS: An Annotation Tool for Long-horizon Robotic Action Segmentationcited by 0
  8. 2026D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 0
  9. 2026ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videoscited by 0
  10. 2026VidParse: Online Parsing of Egocentric Procedures Like a Procited by 0
  11. 2026FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentationcited by 0