Pose-Aware Weakly-Supervised Action Segmentation
Not on the map. Why?
Abstract (arXiv)
Understanding human behavior is an important problem in the pursuit of visual intelligence. A challenge in this endeavor is the extensive and costly effort required to accurately label action segments. To address this issue, we consider learning methods that demand minimal supervision for segmentation of human actions in long instructional videos. Specifically, we introduce a weakly-supervised framework that uniquely incorporates pose knowledge during training while omitting its use during inference, thereby distilling pose knowledge pertinent to each action component. We propose a pose-inspired contrastive loss as a part of the whole weakly-supervised framework which is trained to distinguish action boundaries more effectively. Our approach, validated through extensive experiments on representative datasets, outperforms previous state-of-the-art (SOTA) in segmenting long instructional videos under both online and offline settings. Additionally, we demonstrate the framework's adaptability to various segmentation backbones and pose extractors across different datasets.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 8 Apr 2025.
Builds on
55 of 67 references.
References · 55
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2020Supervised Contrastive Learningcited by 2,829
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
- 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
- 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
- 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
- 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
38 more
- 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 175
- 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
- 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
- 2022Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 85
- 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
- 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
- 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
- 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
- 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
- 2023Learning Discriminative Representations for Skeleton Based Action Recognitioncited by 62
- 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
- 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
- 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 49
- 2014Weakly Supervised Action Labeling in Videos Under Ordering Constraintscited by 48
- 2016Weakly supervised learning of actions from transcriptscited by 46
- 2022Generative Action Description Prompts for Skeleton-based Action Recognitioncited by 43
- 2020Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detectioncited by 42
- 2023Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detectioncited by 42
- 2021VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Livingcited by 35
- 2023RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimationcited by 34
- 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detectioncited by 32
- 2019Fast Weakly Supervised Action Segmentation Using Mutual Consistencycited by 30
- 2019DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognitioncited by 30
- 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
- 2021Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognitioncited by 29
- 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
- 2022PreTraM: Self-Supervised Pre-training via Connecting Trajectory and Mapcited by 25
- 2021Learning Discriminative Prototypes with Dynamic Time Warpingcited by 23
- 2023Prompt-Guided Zero-Shot Anomaly Action Recognition using Pretrained Deep Skeleton Featurescited by 21
- 2022Cross-Modal Learning with 3D Deformable Attention for Action Recognitioncited by 20
- 2020WOAD: Weakly Supervised Online Action Detection in Untrimmed Videoscited by 19
- 2023Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Livingcited by 19
- 2022FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Taskscited by 16
- 2022MovieCLIP: Visual Scene Recognition in Moviescited by 16
- 2023LAC: Latent Action Composition for Skeleton-based Action Segmentationcited by 15
- 2022Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognitioncited by 13
- 2023STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videoscited by 13
- 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.