# DeepSportLab: a Unified Framework for Ball Detection, Player Instance Segmentation and Pose Estimation in Team Sports Scenes

arXiv: https://arxiv.org/abs/2112.00627 · HTML: https://mlsanity.com/paper/2112.00627/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)
- [Towards Accurate Multi-person Pose Estimation in the Wild](https://mlsanity.com/paper/1701.01779/) (2017)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Semantic Instance Segmentation with a Discriminative Loss Function](https://mlsanity.com/paper/1708.02551/) (2017)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [Recurrent Pixel Embedding for Instance Grouping](https://mlsanity.com/paper/1712.08273/) (2017)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation](https://mlsanity.com/paper/1707.05388/) (2017)
- [MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network](https://mlsanity.com/paper/1807.04067/) (2018)
- [Semantic Instance Segmentation via Deep Metric Learning](https://mlsanity.com/paper/1703.10277/) (2017)
- [Semi-convolutional Operators for Instance Segmentation](https://mlsanity.com/paper/1807.10712/) (2018)
- [Proposal-free Network for Instance-level Object Segmentation](https://mlsanity.com/paper/1509.02636/) (2015)
- [Multimodal and multiview distillation for real-time player detection on a football field](https://mlsanity.com/paper/2004.07544/) (2020)
- [Real-time CNN-based Segmentation Architecture for Ball Detection in a Single View Setup](https://mlsanity.com/paper/2007.11876/) (2020)
- [Continuous Video to Simple Signals for Swimming Stroke Detection with Convolutional Neural Networks](https://mlsanity.com/paper/1705.09894/) (2017)

## Built on by (most cited first, 11 of 11)

- [SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap](https://mlsanity.com/paper/2404.11335/) (2024)
- [Multi-task Learning for Joint Re-identification, Team Affiliation, and Role Classification for Sports Visual Tracking](https://mlsanity.com/paper/2401.09942/) (2024)
- [Widely Applicable Strong Baseline for Sports Ball Detection and Tracking](https://mlsanity.com/paper/2311.05237/) (2023)
- [TrackID3x3: A Dataset and Algorithm for Multi-Player Tracking with Identification and Pose Estimation in 3x3 Basketball Full-court Videos](https://mlsanity.com/paper/2503.18282/) (2025)
- [Context-Aware 3D Object Localization from Single Calibrated Images: A Study of Basketballs](https://mlsanity.com/paper/2309.03640/) (2023)
- [Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle](https://mlsanity.com/paper/2412.01562/) (2024)
- [WSESeg: Introducing a Dataset for the Segmentation of Winter Sports Equipment with a Baseline for Interactive Segmentation](https://mlsanity.com/paper/2407.09288/) (2024)
- [SkipClick: Combining Quick Responses and Low-Level Features for Interactive Segmentation in Winter Sports Contexts](https://mlsanity.com/paper/2501.07960/) (2025)
- [Towards Temporal Compositional Reasoning in Long-Form Sports Videos](https://mlsanity.com/paper/2604.22226/) (2026)
- [BBoxMaskPose v2: Expanding Mutual Conditioning to 3D](https://mlsanity.com/paper/2601.15200/) (2026)
- [Scalable Injury-Risk Screening in Baseball Pitching From Broadcast Video](https://mlsanity.com/paper/2603.04864/) (2026)

## Abstract (arXiv)

This paper presents a unified framework to (i) locate the ball, (ii) predict the pose, and (iii) segment the instance mask of players in team sports scenes. Those problems are of high interest in automated sports analytics, production, and broadcast. A common practice is to individually solve each problem by exploiting universal state-of-the-art models, eg, Panoptic-DeepLab for player segmentation. In addition to the increased complexity resulting from the multiplication of single-task models, the use of the off-the-shelf models also impedes the performance due to the complexity and specificity of the team sports scenes, such as strong occlusion and motion blur. To circumvent those limitations, our paper proposes to train a single model that simultaneously predicts the ball and the player mask and pose by combining the part intensity fields and the spatial embeddings principles. Part intensity fields provide the ball and player location, as well as player joints location. Spatial embeddings are then exploited to associate player instance pixels to their respective player center, but also to group player joints into skeletons. We demonstrate the effectiveness of the proposed model on the DeepSport basketball dataset, achieving comparable performance to the SoA models addressing each individual task separately.