mlsanityData to 7 Oct 2026

FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding

arXiv 2004.06704 ↗First posted 14 Apr 2020cs.CVcited by 232

Not on the map. Why?

Abstract (arXiv)

On public benchmarks, current action recognition techniques have achieved great success. However, when used in real-world applications, e.g. sport analysis, which requires the capability of parsing an activity into phases and differentiating between subtly different actions, their performances remain far from being satisfactory. To take action recognition to a new level, we develop FineGym, a new dataset built on top of gymnastic videos. Compared to existing action recognition datasets, FineGym is distinguished in richness, quality, and diversity. In particular, it provides temporal annotations at both action and sub-action levels with a three-level semantic hierarchy. For example, a "balance beam" event will be annotated as a sequence of elementary sub-actions derived from five sets: "leap-jump-hop", "beam-turns", "flight-salto", "flight-handspring", and "dismount", where the sub-action in each set will be further annotated with finely defined class labels. This new level of granularity presents significant challenges for action recognition, e.g. how to parse the temporal structures from a coherent action, and how to distinguish between subtly different action classes. We systematically investigate representative methods on this dataset and obtain a number of interesting findings. We hope this dataset could advance research towards action understanding.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 36; 2023: 35; 2024: 55; 2025: 49; 2026: 32.

232 papers.

Sort:

By area:

  1. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  2. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  3. 2021Revisiting Skeleton-based Action Recognitioncited by 256
  4. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  5. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  6. 2022HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modelingcited by 115
  7. 2022FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessmentcited by 106
  8. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  9. 2022PYSKL: Towards Good Practices for Skeleton Action Recognitioncited by 83
  10. 2021MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actionscited by 82
  11. 2021S2Looking: A Satellite Side-Looking Dataset for Building Change Detectioncited by 77
  12. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
38 more
  1. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  2. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 66
  3. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 60
  4. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  5. 2021Learning the Predictability of the Futurecited by 48
  6. 2024Benchmarking Micro-action Recognition: Dataset, Methods, and Applicationscited by 48
  7. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  8. 2021Representation Learning via Global Temporal Alignment and Cycle-Consistencycited by 43
  9. 2023SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-trainingcited by 41
  10. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  11. 2021SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videoscited by 40
  12. 2022Spotting Temporally Precise, Fine-Grained Events in Videocited by 40
  13. 2024MatchTime: Towards Automatic Soccer Game Commentary Generationcited by 37
  14. 2022TSA-Net: Tube Self-Attention Network for Action Quality Assessmentcited by 36
  15. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  16. 2023M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognitioncited by 34
  17. 2021Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognitioncited by 31
  18. 2024LOGO: A Long-Form Video Dataset for Group Action Quality Assessmentcited by 30
  19. 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
  20. 2021Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognitioncited by 29
  21. 2024Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognitioncited by 29
  22. 2022Frame-wise Action Representations for Long Videos via Sequence Contrastive Learningcited by 27
  23. 2022A Survey on Video Action Recognition in Sports: Datasets, Methods and Applicationscited by 27
  24. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  25. 2024FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessmentcited by 24
  26. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  27. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  28. 2022FLAG3D: A 3D Fitness Activity Dataset with Language Instructioncited by 23
  29. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  30. 2021Self-supervised Motion Learning from Static Imagescited by 21
  31. 2024T-DEED: Temporal-Discriminability Enhancer Encoder-Decoder for Precise Event Spotting in Sports Videoscited by 21
  32. 2024Towards Universal Soccer Video Understandingcited by 21
  33. 2022Cross-Modal Learning with 3D Deformable Attention for Action Recognitioncited by 20
  34. 2024MMAD: Multi-label Micro-Action Detection in Videoscited by 20
  35. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  36. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  37. 2024Narrative Action Evaluation with Prompt-Guided Multimodal Interactioncited by 19
  38. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18

50 most cited of 232 All 232 (JSON)