Is end-to-end learning enough for fitness activity recognition?
Not on the map. Why?
Abstract (arXiv)
End-to-end learning has taken hold of many computer vision tasks, in particular, related to still images, with task-specific optimization yielding very strong performance. Nevertheless, human-centric action recognition is still largely dominated by hand-crafted pipelines, and only individual components are replaced by neural networks that typically operate on individual frames. As a testbed to study the relevance of such pipelines, we present a new fully annotated video dataset of fitness activities. Any recognition capabilities in this domain are almost exclusively a function of human poses and their temporal dynamics, so pose-based solutions should perform well. We show that, with this labelled data, end-to-end learning on raw pixels can compete with state-of-the-art action recognition pipelines based on pose estimation. We also show that end-to-end learning can support temporally fine-grained tasks such as real-time repetition counting.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 14 May 2023.
Builds on
38 of 49 references.
References · 38
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2018SlowFast Networks for Video Recognitioncited by 1,867
- 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
- 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
- 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
- 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
- 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
26 more
- 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
- 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
- 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
- 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
- 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
- 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
- 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
- 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 343
- 2021Revisiting Skeleton-based Action Recognitioncited by 257
- 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
- 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
- 2018Moments in Time Dataset: one million videos for event understandingcited by 215
- 2016Associative Embedding: End-to-End Learning for Joint Detection and Groupingcited by 212
- 2019An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognitioncited by 151
- 2016Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognitioncited by 143
- 2017A New Representation of Skeleton Sequences for 3D Action Recognitioncited by 126
- 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
- 2020BlazePose: On-device Real-time Body Pose trackingcited by 113
- 2021Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regressioncited by 112
- 2017Interpretable 3D Human Action Analysis with Temporal Convolutional Networkscited by 89
- 2019More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregationcited by 64
- 2016Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networkscited by 63
- 2017Skeleton-based Action Recognition with Convolutional Neural Networkscited by 54
- 2018Real-World Repetition Estimation by Div, Grad and Curlcited by 21
- 2018Part-based Graph Convolutional Network for Action Recognitioncited by 19
- 2020NUTA: Non-uniform Temporal Aggregation for Action Recognitioncited by 6
Built on by
Citing papers per year: 2024: 1.
1 paper.