mlsanityData to 7 Oct 2026

MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions

arXiv 2105.07404 ↗First posted 16 May 2021cs.CVcited by 82

Not on the map. Why?

Abstract (arXiv)

Spatio-temporal action detection is an important and challenging problem in video understanding. The existing action detection benchmarks are limited in aspects of small numbers of instances in a trimmed video or low-level atomic actions. This paper aims to present a new multi-person dataset of spatio-temporal localized sports actions, coined as MultiSports. We first analyze the important ingredients of constructing a realistic and challenging dataset for spatio-temporal action detection by proposing three criteria: (1) multi-person scenes and motion dependent identification, (2) with well-defined boundaries, (3) relatively fine-grained classes of high complexity. Based on these guide-lines, we build the dataset of MultiSports v1.0 by selecting 4 sports classes, collecting 3200 video clips, and annotating 37701 action instances with 902k bounding boxes. Our datasets are characterized with important properties of high diversity, dense annotation, and high quality. Our Multi-Sports, with its realistic setting and detailed annotations, exposes the intrinsic challenges of spatio-temporal action detection. To benchmark this, we adapt several baseline methods to our dataset and give an in-depth analysis on the action detection results in our dataset. We hope our MultiSports can serve as a standard benchmark for spatio-temporal action detection in the future. Our dataset website is at https://deeperaction.github.io/multisports/.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 16 May 2021.

Builds on

49 of 68 references.

References · 49

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  10. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
37 more
  1. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  2. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  3. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  4. 2017Deep Layer Aggregationcited by 436
  5. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  6. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  7. 2018Video Action Transformer Networkcited by 316
  8. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  9. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  10. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  11. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  12. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  13. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  14. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  15. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  16. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  17. 2020The AVA-Kinetics Localized Human Actions Video Datasetcited by 82
  18. 2017Action Tubelet Detector for Spatio-Temporal Action Localizationcited by 77
  19. 2020Asynchronous Interaction Aggregation for Action Detectioncited by 65
  20. 2017Appearance-and-Relation Networks for Video Classificationcited by 61
  21. 2015Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptorscited by 58
  22. 2019You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localizationcited by 58
  23. 2016Online Real-time Multiple Spatiotemporal Action Localisation and Predictioncited by 57
  24. 2020Actions as Moving Pointscited by 55
  25. 2019STEP: Spatio-Temporal Progressive Learning for Video Action Detectioncited by 51
  26. 2014Finding Action Tubescited by 50
  27. 2017Tube Convolutional Neural Network (T-CNN) for Action Detection in Videoscited by 50
  28. 2015Learning to track for spatio-temporal action localizationcited by 44
  29. 2020MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detectioncited by 44
  30. 2019Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understandingcited by 37
  31. 2019TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detectioncited by 36
  32. 2016Deep Learning for Detecting Multiple Space-Time Action Tubes in Videoscited by 35
  33. 2020Context-Aware RCNN: A Baseline for Action Detection in Videoscited by 33
  34. 2019Dance with Flow: Two-in-One Stream Action Detectioncited by 30
  35. 2016Actionness Estimation Using Hybrid Fully Convolutional Networkscited by 15
  36. 2020Fully Convolutional Online Trackingcited by 15
  37. 2016Human Action Localization with Sparse Spatial Supervisioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 23; 2024: 15; 2025: 19; 2026: 12.

82 papers.

Sort:

By area:

  1. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  2. 2023Video Understanding with Large Language Models: A Surveycited by 203
  3. 2023SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenescited by 130
  4. 2022FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessmentcited by 106
  5. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 60
  6. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  7. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  8. 2022Spotting Temporally Precise, Fine-Grained Events in Videocited by 40
  9. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  10. 2024LOGO: A Long-Form Video Dataset for Group Action Quality Assessmentcited by 30
  11. 2022Holistic Interaction Transformer Network for Action Detectioncited by 28
  12. 2022A Survey on Video Action Recognition in Sports: Datasets, Methods and Applicationscited by 27
38 more
  1. 2024FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessmentcited by 24
  2. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  3. 2023STMixer: A One-Stage Sparse Action Detectorcited by 22
  4. 2023End-to-End Spatio-Temporal Action Localisation with Video Transformerscited by 20
  5. 2023MotionTrack: Learning Motion Predictor for Multiple Object Trackingcited by 20
  6. 2023NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understandingcited by 20
  7. 2023Efficient Video Action Detection with Token Dropout and Context Refinementcited by 18
  8. 2024Dual DETRs for Multi-Label Temporal Action Detectioncited by 18
  9. 2025BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimationcited by 17
  10. 2023TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognitioncited by 16
  11. 2022Spatio-Temporal Action Detection Under Large Motioncited by 14
  12. 2024SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videoscited by 14
  13. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  14. 2025Video Action Differencingcited by 14
  15. 2022Panoramic Human Activity Recognitioncited by 13
  16. 2022Cross-Architecture Self-supervised Video Representation Learningcited by 13
  17. 2023ShuttleSet: A Human-Annotated Stroke-Level Singles Dataset for Badminton Tactical Analysiscited by 13
  18. 2023A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decisioncited by 11
  19. 2024Computer Vision for Primate Behavior Analysis in the Wildcited by 10
  20. 2024Language and Multimodal Models in Sports: A Survey of Datasets and Applicationscited by 10
  21. 2025SoccerMaster: A Vision Foundation Model for Soccer Understandingcited by 10
  22. 2024ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognitioncited by 8
  23. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detectioncited by 8
  24. 2025ExAct: A Video-Language Benchmark for Expert Action Analysiscited by 8
  25. 2023Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer -- Current Trends and Research Perspectivescited by 7
  26. 2022A Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detectorcited by 6
  27. 2024Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Videocited by 6
  28. 2025Game State and Spatio-temporal Action Detection in Soccer using Graph Neural Networks and 3D Convolutional Networkscited by 6
  29. 2025MOVE: Motion-Guided Few-Shot Video Object Segmentationcited by 6
  30. 2023A Survey on Deep Learning-based Spatio-temporal Action Detectioncited by 5
  31. 2025Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videoscited by 5
  32. 2025FOOTPASS: A Multi-Modal Multi-Agent Tactical Context Dataset for Play-by-Play Action Spotting in Soccer Broadcast Videoscited by 5
  33. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Eventscited by 5
  34. 2022Fine-grained Activities of People Worldwidecited by 4
  35. 2025SGA-INTERACT: A 3D Skeleton-based Benchmark for Group Activity Understanding in Modern Basketball Tacticcited by 4
  36. 2025Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videoscited by 4
  37. 2023Real-time Multi-person Eyeblink Detection in the Wild for Untrimmed Videocited by 3
  38. 2023CycleACR: Cycle Modeling of Actor-Context Relations for Video Action Detectioncited by 3

50 most cited of 82 All 82 (JSON)