mlsanityData to 14 Oct 2026

Temporal Action Segmentation: An Analysis of Modern Techniques

arXiv 2210.10352 ↗First posted 19 Oct 2022cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Temporal action segmentation (TAS) in videos aims at densely identifying video frames in minutes-long videos with multiple action classes. As a long-range video understanding task, researchers have developed an extended collection of methods and examined their performance using various benchmarks. Despite the rapid growth of TAS techniques in recent years, no systematic survey has been conducted in these sectors. This survey analyzes and summarizes the most significant contributions and trends. In particular, we first examine the task definition, common benchmarks, types of supervision, and prevalent evaluation measures. In addition, we systematically investigate two essential techniques of this topic, i.e., frame representation and temporal modeling, which have been studied extensively in the literature. We then conduct a thorough review of existing TAS works categorized by their levels of supervision and conclude our survey by identifying and emphasizing several research gaps. In addition, we have curated a list of TAS resources, which is available at https://github.com/nus-cvml/awesome-temporal-action-segmentation.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 122 of its 206 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

122 of 206 references.

References · 122

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
38 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,867
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  7. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  8. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  10. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  11. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  13. 2020Rescaling Egocentric Visioncited by 540
  14. 2018Object Detection with Deep Learning: A Reviewcited by 528
  15. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  16. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  17. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  18. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  19. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  20. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 329
  21. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  22. 2018Video Action Transformer Networkcited by 316
  23. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  24. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  25. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  26. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  27. 2022Egocentric Video-Language Pretrainingcited by 232
  28. 2018Human Action Recognition and Prediction: A Surveycited by 227
  29. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  30. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  31. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  32. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  33. 2021ASFormer: Transformer for Action Segmentationcited by 168
  34. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  35. 2016Video Summarization with Long Short-term Memorycited by 142
  36. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  37. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  38. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 19; 2025: 36; 2026: 15.

85 papers.

Sort:

By area:

  1. 2023MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensingcited by 79
  2. 2023Diffusion Action Segmentationcited by 74
  3. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 38
  4. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 36
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  6. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  7. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  8. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  9. 2023Every Mistake Counts in Assemblycited by 18
  10. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  11. 2023Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairscited by 17
  12. 2023OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentationcited by 11
38 more
  1. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  2. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesiscited by 11
  3. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  4. 2023Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transportcited by 10
  5. 2023DiffAnt: Diffusion Models for Action Anticipationcited by 9
  6. 2024ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understandingcited by 8
  7. 20243D Pose-Based Temporal Action Segmentation for Figure Skating: A Fine-Grained and Jump Procedure-Aware Annotation Approachcited by 7
  8. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  9. 2025F3Set: Towards Analyzing Fast, Frequent, and Fine-grained Events from Videoscited by 7
  10. 2023TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognitioncited by 6
  11. 2023Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusioncited by 6
  12. 2023Deep Learning Approaches for Seizure Video Analysis: A Reviewcited by 6
  13. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  14. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  15. 2024Coherent Temporal Synthesis for Incremental Action Segmentationcited by 5
  16. 2025Recognizing Ornaments in Vocal Indian Art Music with Active Annotationcited by 5
  17. 2025ProMQA-Assembly: Multimodal Procedural QA Dataset on Assemblycited by 5
  18. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  19. 2024Real-Time Human Action Recognition on Embedded Platformscited by 4
  20. 2024ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipationcited by 4
  21. 2025Condensing Action Segmentation Datasets via Generative Network Inversioncited by 4
  22. 2025PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experimentscited by 4
  23. 2023A Video-based End-to-end Pipeline for Non-nutritive Sucking Action Recognition and Segmentation in Young Infantscited by 3
  24. 2024Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustmentcited by 3
  25. 20242by2: Weakly-Supervised Learning for Global Action Segmentationcited by 3
  26. 2025Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modelingcited by 3
  27. 2024Learning to Visually Connect Actions and their Effectscited by 2
  28. 2024SuPRA: Surgical Phase Recognition and Anticipation for Intra-Operative Planningcited by 2
  29. 2024Con4m: Context-aware Consistency Learning Framework for Segmented Time Series Classificationcited by 2
  30. 2024ChildPlay-Hand: A Dataset of Hand Manipulations in the Wildcited by 2
  31. 2024Language-Assisted Human Part Motion Learning for Skeleton-Based Temporal Action Segmentationcited by 2
  32. 2025Text-Derived Relational Graph-Enhanced Network for Skeleton-Based Action Segmentationcited by 2
  33. 2025Generic Event Boundary Detection via Denoising Diffusioncited by 2
  34. 2026ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarioscited by 2
  35. 2023End-to-End Streaming Video Temporal Action Segmentation with Reinforce Learningcited by 1
  36. 2023Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domainscited by 1
  37. 2024Wormhole: Concept-Aware Deep Representation Learning for Co-Evolving Sequencescited by 1
  38. 2025A Temporal Convolutional Network-Based Approach and a Benchmark Dataset for Colonoscopy Video Temporal Segmentationcited by 1

50 most cited of 85 All 85 (JSON)