Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors
Not on the map. Why?
Abstract (arXiv)
Sports analysis and viewing play a pivotal role in the current sports domain, offering significant value not only to coaches and athletes but also to fans and the media. In recent years, the rapid development of virtual reality (VR) and augmented reality (AR) technologies have introduced a new platform for watching games. Visualization of sports competitions in VR/AR represents a revolutionary technology, providing audiences with a novel immersive viewing experience. However, there is still a lack of related research in this area. In this work, we present for the first time a comprehensive system for sports competition analysis and real-time visualization on VR/AR platforms. First, we utilize multiview LiDARs and cameras to collect multimodal game data. Subsequently, we propose a framework for multi-player tracking and pose estimation based on a limited amount of supervised data, which extracts precise player positions and movements from point clouds and images. Moreover, we perform avatar modeling of players to obtain their 3D models. Ultimately, using these 3D player data, we conduct competition analysis and real-time visualization on VR/AR. Extensive quantitative experiments demonstrate the accuracy and robustness of our multi-player tracking and pose estimation framework. The visualization results showcase the immense potential of our sports visualization system on the domain of watching games on VR/AR devices. The multimodal competition dataset we collected and all related code will be released soon.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 47 of its 76 references that are in the map. First posted on arXiv 2 May 2024.
Builds on
47 of 76 references.
References · 47
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2017Mask R-CNNcited by 7,259
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2015SSD: Single Shot MultiBox Detectorcited by 3,125
- 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
- 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
- 2020Center-based 3D Object Detection and Trackingcited by 1,143
- 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
- 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
- 2020Tracking Objects as Pointscited by 483
- 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 454
35 more
- 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
- 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
- 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
- 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
- 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 199
- 2020Whole-Body Human Pose Estimation in the Wildcited by 178
- 2019Learnable Triangulation of Human Posecited by 160
- 2022SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Videocited by 144
- 2020GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learningcited by 113
- 2018Pose2Seg: Detection Free Human Instance Segmentationcited by 107
- 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
- 2019Self-Supervised Learning of 3D Human Pose using Multi-view Geometrycited by 98
- 2019Fast and Robust Multi-Person 3D Pose Estimation from Multiple Viewscited by 83
- 2020Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Drivingcited by 73
- 2021Direct Multi-view Multi-person 3D Pose Estimationcited by 64
- 2020Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representationcited by 57
- 20214D-Net for Learned Multi-Modal Alignmentcited by 44
- 2021VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wildcited by 39
- 2021Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Drivingcited by 39
- 2022Faster VoxelPose: Real-time 3D Human Pose Estimation by Orthographic Projectioncited by 39
- 2021Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereocited by 38
- 2021Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Imagescited by 38
- 2019Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Cloudscited by 35
- 2021LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Trackingcited by 34
- 2022LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Cloudscited by 32
- 2021DyGLIP: A Dynamic Graph Model with Link Prediction for Accurate Multi-Camera Multiple Object Trackingcited by 31
- 2022Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimationcited by 25
- 2021DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoderscited by 23
- 2022Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDARcited by 23
- 2020PointTrackNet: An End-to-End Network For 3-D Object Detection and Tracking From Point Cloudscited by 20
- 2021Real-time RGBD-based Extended Body Pose Estimationcited by 12
- 2017Towards Good Practices for Deep 3D Hand Pose Estimationcited by 11
- 20223D Human Pose Estimation in Multi-View Operating Room Videos Using Differentiable Camera Projectionscited by 7
- 2020Residual Pose: A Decoupled Approach for Depth-based 3D Human Pose Estimationcited by 5
- 2020View Invariant Human Body Detection and Pose Estimation from Multiple Depth Sensorscited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.