# Syn4D: A Multiview Synthetic 4D Dataset

arXiv: https://arxiv.org/abs/2605.05207 · HTML: https://mlsanity.com/paper/2605.05207/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 113 of its 137 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 30 of 113)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)

## Built on by (most cited first, 4 of 4)

- [WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations](https://mlsanity.com/paper/2608.15659/) (2026)
- [TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views](https://mlsanity.com/paper/2609.01899/) (2026)
- [Gallileo-4D: Frozen Backbone Ensemble for Dynamic 4D Reconstruction](https://mlsanity.com/paper/2608.19743/) (2026)
- [ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild](https://mlsanity.com/paper/2610.01314/) (2026)

## Abstract (arXiv)

Dense 3D reconstruction and tracking of dynamic scenes from monocular video remains an important open challenge in computer vision. Progress in this area has been constrained by the scarcity of high-quality datasets with dense, complete, and accurate geometric annotations. To address this limitation, we introduce Syn4D, a multiview synthetic dataset of dynamic scenes that includes ground-truth camera motion, depth maps, dense tracking, and parametric human pose annotations. A key feature of Syn4D is the ability to unproject any pixel into 3D to any time and to any camera. We conduct extensive evaluations across multiple downstream tasks to demonstrate the utility and effectiveness of the proposed dataset, including 4D scene reconstruction, 3D point tracking, geometry-aware camera retargeting, and human pose estimation. The experimental results highlight Syn4D's potential to facilitate research in dynamic scene understanding and spatiotemporal modeling.