# Black-Box vs. Gray-Box: A Case Study on Learning Table Tennis Ball Trajectory Prediction with Spin and Impacts

arXiv: https://arxiv.org/abs/2305.15189 · HTML: https://mlsanity.com/paper/2305.15189/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces](https://mlsanity.com/paper/1905.07357/) (2019)
- [Real Time Trajectory Prediction Using Deep Conditional Generative Models](https://mlsanity.com/paper/1909.03895/) (2019)
- [Reliable Real Time Ball Tracking for Robot Table Tennis](https://mlsanity.com/paper/1908.07332/) (2019)
- [Learning to Control Highly Accelerated Ballistic Movements on Muscular Robots](https://mlsanity.com/paper/1904.03665/) (2019)
- [AIMY: An Open-source Table Tennis Ball Launcher for Versatile and High-fidelity Trajectory Generation](https://mlsanity.com/paper/2210.06048/) (2022)

## Built on by (most cited first, 4 of 4)

- [Table tennis ball spin estimation with an event camera](https://mlsanity.com/paper/2404.09870/) (2024)
- [Towards Ball Spin and Trajectory Analysis in Table Tennis Broadcast Videos via Physically Grounded Synthetic-to-Real Transfer](https://mlsanity.com/paper/2504.19863/) (2025)
- [Egocentric Event-Based Vision for Ping Pong Ball Trajectory Prediction](https://mlsanity.com/paper/2506.07860/) (2025)
- [BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs](https://mlsanity.com/paper/2605.30900/) (2026)

## Abstract (arXiv)

In this paper, we present a method for table tennis ball trajectory filtering and prediction. Our gray-box approach builds on a physical model. At the same time, we use data to learn parameters of the dynamics model, of an extended Kalman filter, and of a neural model that infers the ball's initial condition. We demonstrate superior prediction performance of our approach over two black-box approaches, which are not supplied with physical prior knowledge. We demonstrate that initializing the spin from parameters of the ball launcher using a neural network drastically improves long-time prediction performance over estimating the spin purely from measured ball positions. An accurate prediction of the ball trajectory is crucial for successful returns. We therefore evaluate the return performance with a pneumatic artificial muscular robot and achieve a return rate of 29/30 (97.7%).