mlsanityData to 14 Oct 2026

Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusion

arXiv 2309.06462 ↗First posted 12 Sep 2023cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents a 2D skeleton-based action segmentation method with applications in fine-grained human activity recognition. In contrast with state-of-the-art methods which directly take sequences of 3D skeleton coordinates as inputs and apply Graph Convolutional Networks (GCNs) for spatiotemporal feature learning, our main idea is to use sequences of 2D skeleton heatmaps as inputs and employ Temporal Convolutional Networks (TCNs) to extract spatiotemporal features. Despite lacking 3D information, our approach yields comparable/superior performances and better robustness against missing keypoints than previous methods on action segmentation datasets. Moreover, we improve the performances further by using both 2D skeleton heatmaps and RGB videos as inputs. To our best knowledge, this is the first work to utilize 2D skeleton heatmap inputs and the first work to explore 2D skeleton+RGB fusion for action segmentation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

28 of 43 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Fast R-CNNcited by 2,933
  4. 2018SlowFast Networks for Video Recognitioncited by 1,867
  5. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  6. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  7. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  8. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  9. 2014Deeply-Supervised Netscited by 367
  10. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  11. 2021Revisiting Skeleton-based Action Recognitioncited by 257
  12. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
16 more
  1. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  2. 2020Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognitioncited by 91
  3. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  4. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  5. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 49
  6. 2019SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognitioncited by 43
  7. 2020JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognitioncited by 33
  8. 2017Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognitioncited by 23
  9. 2022Skeleton-Based Action Segmentation with Multi-Stage Spatial-Temporal Graph Convolutional Neural Networkscited by 20
  10. 2016Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsingcited by 19
  11. 2018Deep Supervision with Intermediate Conceptscited by 19
  12. 2022Timestamp-Supervised Action Segmentation with Graph Convolutional Networkscited by 18
  13. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  14. 2017TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentationcited by 14
  15. 2019Spatio-Temporal Pyramid Graph Convolutions for Human Action Recognition and Postural Assessmentcited by 9
  16. 2019Toward Ergonomic Risk Prediction via Segmentation of Indoor Object Manipulation Actions Using Spatiotemporal Convolutional Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  2. 2025Foundation Model for Skeleton-Based Human Action Understandingcited by 11
  3. 2025Procedure Learning via Regularized Gromov-Wasserstein Optimal Transportcited by 4
  4. 2025Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulationcited by 3
  5. 2025Biomechanical-phase based Temporal Segmentation in Sports Videos: a Demonstration on Javelin-Throwcited by 1
  6. 2026Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantizationcited by 0