# TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos

arXiv: https://arxiv.org/abs/2605.01234 · HTML: https://mlsanity.com/paper/2605.01234/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 2 May 2026.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Simulation-free Schr"odinger bridges via score and flow matching](https://mlsanity.com/paper/2307.03672/) (2023)
- [Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking](https://mlsanity.com/paper/2510.02252/) (2025)
- [TTNet: Real-time temporal and spatial video analysis of table tennis](https://mlsanity.com/paper/2004.09927/) (2020)
- [TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications](https://mlsanity.com/paper/1907.03698/) (2019)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [HITTER: A HumanoId Table TEnnis Robot via Hierarchical Planning and Learning](https://mlsanity.com/paper/2508.21043/) (2025)
- [MonoTrack: Shuttle trajectory reconstruction from monocular badminton video](https://mlsanity.com/paper/2204.01899/) (2022)
- [Table Tennis Stroke Recognition Using Two-Dimensional Human Pose Estimation](https://mlsanity.com/paper/2104.09907/) (2021)
- [DeepBall: Deep Neural-Network Ball Detector](https://mlsanity.com/paper/1902.07304/) (2019)
- [Table tennis ball spin estimation with an event camera](https://mlsanity.com/paper/2404.09870/) (2024)
- [TrackNetV4: Enhancing Fast Sports Object Tracking with Motion Attention Maps](https://mlsanity.com/paper/2409.14543/) (2024)
- [Spin Detection in Robotic Table Tennis](https://mlsanity.com/paper/1905.07967/) (2019)
- [Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2405.14467/) (2024)
- [Real-time CNN-based Segmentation Architecture for Ball Detection in a Single View Setup](https://mlsanity.com/paper/2007.11876/) (2020)
- [Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data](https://mlsanity.com/paper/2603.12686/) (2026)
- [P2ANet: A Dataset and Benchmark for Dense Action Detection from Table Tennis Match Broadcasting Videos](https://mlsanity.com/paper/2207.12730/) (2022)
- [LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos](https://mlsanity.com/paper/2503.20936/) (2025)
- [AIMY: An Open-source Table Tennis Ball Launcher for Versatile and High-fidelity Trajectory Generation](https://mlsanity.com/paper/2210.06048/) (2022)
- [Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation](https://mlsanity.com/paper/2511.20250/) (2025)
- [Where Is The Ball: 3D Ball Trajectory Estimation From 2D Monocular Tracking](https://mlsanity.com/paper/2506.05763/) (2025)

## Built on by (most cited first, 1 of 1)

- [Rule-Constrained Assignment for Cue-Ball Identification in Broadcast Snooker](https://mlsanity.com/paper/2609.23450/) (2026)

## Abstract (arXiv)

We present TT4D, a large-scale, high-fidelity table tennis dataset. It provides 140+ hours of reconstructed singles and doubles gameplay from monocular broadcast videos, featuring multimodal annotations like high-quality camera calibrations, precise 3D ball positions, ball spin, time segmentation, and 3D human meshes over time. This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning. The dataset's combination of scale and precision is achieved through a novel reconstruction pipeline. Prior methods first partition a game sequence into individual shot segments based on the 2D ball track, and only then attempt reconstruction. However, 2D-based time segmentation collapses under occlusion and varied camera viewpoints, preventing reliable reconstruction. We invert this paradigm by first lifting the entire unsegmented 2D ball track to 3D through a learned lifting network. This 3D trajectory then allows us to reliably perform time segmentation. The learned lifting network also infers the ball's spin, handles unreliable ball detections, and successfully reconstructs the ball trajectory in cases of high occlusion. This lift-first design is necessary, as our pipeline is the only method capable of reconstructing table tennis gameplay from general-view broadcast monocular videos. We demonstrate the dataset's fidelity through two downstream tasks: estimating the racket's pose & velocity at impact, and training a generative model of competitive rallies.