# Multi-Camera Asynchronous Ball Localization and Trajectory Prediction with Factor Graphs and Human Poses

arXiv: https://arxiv.org/abs/2401.17185 · HTML: https://mlsanity.com/paper/2401.17185/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrians](https://mlsanity.com/paper/1805.11833/) (2018)
- [YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers](https://mlsanity.com/paper/1811.05588/) (2018)
- [i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops](https://mlsanity.com/paper/2207.06572/) (2022)
- [Table Tennis Stroke Recognition Using Two-Dimensional Human Pose Estimation](https://mlsanity.com/paper/2104.09907/) (2021)
- [Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation](https://mlsanity.com/paper/2001.00503/) (2020)
- [Ball 3D Localization From A Single Calibrated Image](https://mlsanity.com/paper/2204.00003/) (2022)
- [Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2209.11908/) (2022)
- [Robotic Table Tennis: A Case Study into a High Speed Learning System](https://mlsanity.com/paper/2309.03315/) (2023)
- [Real Time Trajectory Prediction Using Deep Conditional Generative Models](https://mlsanity.com/paper/1909.03895/) (2019)
- [Tracking Small and Fast Moving Objects: A Benchmark](https://mlsanity.com/paper/2209.04284/) (2022)
- [Spin Detection in Robotic Table Tennis](https://mlsanity.com/paper/1905.07967/) (2019)
- [Athletic Mobile Manipulator System for Robotic Wheelchair Tennis](https://mlsanity.com/paper/2210.02517/) (2022)
- [SpinDOE: A ball spin estimation method for table tennis robot](https://mlsanity.com/paper/2303.03879/) (2023)
- [Reliable Real Time Ball Tracking for Robot Table Tennis](https://mlsanity.com/paper/1908.07332/) (2019)
- [Table Tennis Stroke Detection and Recognition Using Ball Trajectory Data](https://mlsanity.com/paper/2302.09657/) (2023)
- [The Effect of Robot Skill Level and Communication in Rapid, Proximate Human-Robot Collaboration](https://mlsanity.com/paper/2304.03756/) (2023)

## Built on by (most cited first, 4 of 4)

- [Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidance](https://mlsanity.com/paper/2409.15528/) (2024)
- [Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control](https://mlsanity.com/paper/2504.17771/) (2025)
- [Towards Ball Spin and Trajectory Analysis in Table Tennis Broadcast Videos via Physically Grounded Synthetic-to-Real Transfer](https://mlsanity.com/paper/2504.19863/) (2025)
- [Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation](https://mlsanity.com/paper/2511.20250/) (2025)

## Abstract (arXiv)

The rapid and precise localization and prediction of a ball are critical for developing agile robots in ball sports, particularly in sports like tennis characterized by high-speed ball movements and powerful spins. The Magnus effect induced by spin adds complexity to trajectory prediction during flight and bounce dynamics upon contact with the ground. In this study, we introduce an innovative approach that combines a multi-camera system with factor graphs for real-time and asynchronous 3D tennis ball localization. Additionally, we estimate hidden states like velocity and spin for trajectory prediction. Furthermore, to enhance spin inference early in the ball's flight, where limited observations are available, we integrate human pose data using a temporal convolutional network (TCN) to compute spin priors within the factor graph. This refinement provides more accurate spin priors at the beginning of the factor graph, leading to improved early-stage hidden state inference for prediction. Our result shows the trained TCN can predict the spin priors with RMSE of 5.27 Hz. Integrating TCN into the factor graph reduces the prediction error of landing positions by over 63.6% compared to a baseline method that utilized an adaptive extended Kalman filter.