# LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos

arXiv: https://arxiv.org/abs/2503.20936 · HTML: https://mlsanity.com/paper/2503.20936/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Uncertainty Quantification and Deep Ensembles](https://mlsanity.com/paper/2007.08792/) (2020)
- [Humanoid Locomotion as Next Token Prediction](https://mlsanity.com/paper/2402.19469/) (2024)
- [Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology](https://mlsanity.com/paper/2204.04516/) (2022)
- [TTNet: Real-time temporal and spatial video analysis of table tennis](https://mlsanity.com/paper/2004.09927/) (2020)
- [Learning to Play Table Tennis From Scratch using Muscular Robots](https://mlsanity.com/paper/2006.05935/) (2020)
- [i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops](https://mlsanity.com/paper/2207.06572/) (2022)
- [Achieving Human Level Competitive Robot Table Tennis](https://mlsanity.com/paper/2408.03906/) (2024)
- [Robotic Table Tennis with Model-Free Reinforcement Learning](https://mlsanity.com/paper/2003.14398/) (2020)
- [Sample-efficient Reinforcement Learning in Robotic Table Tennis](https://mlsanity.com/paper/2011.03275/) (2020)
- [GoalsEye: Learning High Speed Precision Table Tennis on a Physical Robot](https://mlsanity.com/paper/2210.03662/) (2022)

## Built on by (most cited first, 5 of 5)

- [Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation](https://mlsanity.com/paper/2511.20250/) (2025)
- [RacketVision: A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis](https://mlsanity.com/paper/2511.17045/) (2025)
- [BlurBall: Joint Ball and Motion Blur Estimation for Table Tennis Ball Tracking](https://mlsanity.com/paper/2509.18387/) (2025)
- [TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos](https://mlsanity.com/paper/2605.01234/) (2026)
- [How good was my shot? Quantifying Player Skill Level in Table Tennis](https://mlsanity.com/paper/2603.25736/) (2026)

## Abstract (arXiv)

Physical agility is a necessary skill in competitive table tennis, but by no means sufficient. Champions excel in this fast-paced and highly dynamic environment by anticipating their opponent's intent - buying themselves the necessary time to react. In this work, we take one step towards designing such an anticipatory agent. Previous works have developed systems capable of real-time table tennis gameplay, though they often do not leverage anticipation. Among the works that forecast opponent actions, their approaches are limited by dataset size and variety. Our paper contributes (1) a scalable system for reconstructing monocular video of table tennis matches in 3D and (2) an uncertainty-aware controller that anticipates opponent actions. We demonstrate in simulation that our policy improves the ball return rate against high-speed hits from 49.9% to 59.0% as compared to a baseline non-anticipatory policy.