# Pose2Trajectory: Using Transformers on Body Pose to Predict Tennis Player's Trajectory

arXiv: https://arxiv.org/abs/2411.04501 · HTML: https://mlsanity.com/paper/2411.04501/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs](https://mlsanity.com/paper/1904.09507/) (2019)
- [TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications](https://mlsanity.com/paper/1907.03698/) (2019)
- [S2TNet: Spatio-Temporal Transformer Networks for Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2206.10902/) (2022)
- [PreTR: Spatio-Temporal Non-Autoregressive Trajectory Prediction Transformer](https://mlsanity.com/paper/2203.09293/) (2022)
- [Memory Augmented Deep Generative models for Forecasting the Next Shot Location in Tennis](https://mlsanity.com/paper/1901.05123/) (2019)

## Built on by (most cited first, 5 of 5)

- [SMART-Vision: Survey of Modern Action Recognition Techniques in Vision](https://mlsanity.com/paper/2501.13066/) (2025)
- [SGNetPose+: Stepwise Goal-Driven Networks with Pose Information for Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.08016/) (2025)
- [Multi-person Physics-based Pose Estimation for Combat Sports](https://mlsanity.com/paper/2504.08175/) (2025)
- [2COOOL: 2nd Workshop on the Challenge Of Out-Of-Label Hazards in Autonomous Driving](https://mlsanity.com/paper/2508.21080/) (2025)
- [BFMD: A Full-Match Badminton Dense Dataset for Dense Shot Captioning](https://mlsanity.com/paper/2603.25533/) (2026)

## Abstract (arXiv)

Tracking the trajectory of tennis players can help camera operators in production. Predicting future movement enables cameras to automatically track and predict a player's future trajectory without human intervention. Predicting future human movement in the context of complex physical tasks is also intellectually satisfying. Swift advancements in sports analytics and the wide availability of videos for tennis have inspired us to propose a novel method called Pose2Trajectory, which predicts a tennis player's future trajectory as a sequence derived from their body joints' data and ball position. Demonstrating impressive accuracy, our approach capitalizes on body joint information to provide a comprehensive understanding of the human body's geometry and motion, thereby enhancing the prediction of the player's trajectory. We use encoder-decoder Transformer architecture trained on the joints and trajectory information of the players with ball positions. The predicted sequence can provide information to help close-up cameras to keep tracking the tennis player, following centroid coordinates. We generate a high-quality dataset from multiple videos to assist tennis player movement prediction using object detection and human pose estimation methods. It contains bounding boxes and joint information for tennis players and ball positions in singles tennis games. Our method shows promising results in predicting the tennis player's movement trajectory with different sequence prediction lengths using the joints and trajectory information with the ball position.