mlsanityData to 14 Oct 2026

Towards Generalizing Temporal Action Segmentation to Unseen Views

arXiv 2504.02512 ↗First posted 3 Apr 2025cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

While there has been substantial progress in temporal action segmentation, the challenge to generalize to unseen views remains unaddressed. Hence, we define a protocol for unseen view action segmentation where camera views for evaluating the model are unavailable during training. This includes changing from top-frontal views to a side view or even more challenging from exocentric to egocentric views. Furthermore, we present an approach for temporal action segmentation that tackles this challenge. Our approach leverages a shared representation at both the sequence and segment levels to reduce the impact of view differences during training. We achieve this by introducing a sequence loss and an action loss, which together facilitate consistent video and action representations across different views. The evaluation on the Assembly101, IkeaASM, and EgoExoLearn datasets demonstrate significant improvements, with a 12.8% increase in F1@50 for unseen exocentric views and a substantial 54% improvement for unseen egocentric views.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 62 of its 98 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

62 of 98 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  12. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
38 more
  1. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  2. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 568
  3. 2020Rescaling Egocentric Visioncited by 540
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  5. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  6. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  7. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  8. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 343
  9. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  10. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  11. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  12. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  13. 2021ASFormer: Transformer for Action Segmentationcited by 168
  14. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 128
  15. 2017View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Datacited by 105
  16. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 103
  17. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  18. 2020Skeleton-based Action Recognition via Spatial and Temporal Transformer Networkscited by 98
  19. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  20. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  21. 2017Interpretable 3D Human Action Analysis with Temporal Convolutional Networkscited by 89
  22. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  23. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  24. 2020Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognitioncited by 82
  25. 2023Diffusion Action Segmentationcited by 74
  26. 2022Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognitioncited by 71
  27. 2015A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestioncited by 68
  28. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  29. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  30. 2017Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gatescited by 54
  31. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50
  32. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  33. 2017Unsupervised Learning of Long-Term Motion Dynamics for Videoscited by 37
  34. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  35. 2021UNIK: A Unified Framework for Real-world Skeleton-based Action Recognitioncited by 35
  36. 2018Unsupervised Learning of View-invariant Action Representationscited by 32
  37. 2021ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpointscited by 20
  38. 2023Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognitioncited by 20

50 most cited of 62 All 62 (JSON)

Built on by

Not cited yet.