# PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking

arXiv: https://arxiv.org/abs/2307.15055 · HTML: https://mlsanity.com/paper/2307.15055/

## Key facts

- Cited by 276 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)

## Built on by (most cited first, 30 of 276)

- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Shape of Motion: 4D Reconstruction from a Single Video](https://mlsanity.com/paper/2407.13764/) (2024)
- [Visual Geometry Grounded Deep Structure From Motion](https://mlsanity.com/paper/2312.04563/) (2023)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models](https://mlsanity.com/paper/2411.18613/) (2024)
- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [Magma: A Foundation Model for Multimodal AI Agents](https://mlsanity.com/paper/2502.13130/) (2025)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models](https://mlsanity.com/paper/2503.05638/) (2025)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis](https://mlsanity.com/paper/2405.14868/) (2024)
- [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025)
- [Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos](https://mlsanity.com/paper/2412.09621/) (2024)
- [St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World](https://mlsanity.com/paper/2504.13152/) (2025)
- [Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory](https://mlsanity.com/paper/2507.02863/) (2025)
- [SpatialTrackerV2: 3D Point Tracking Made Easy](https://mlsanity.com/paper/2507.12462/) (2025)
- [Local All-Pair Correspondence for Point Tracking](https://mlsanity.com/paper/2407.15420/) (2024)
- [Align3R: Aligned Monocular Depth Estimation for Dynamic Videos](https://mlsanity.com/paper/2412.03079/) (2024)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [TAPVid-3D: A Benchmark for Tracking Any Point in 3D](https://mlsanity.com/paper/2407.05921/) (2024)

## Abstract (arXiv)

We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks. Our data and code are publicly available at: https://pointodyssey.com