# MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data

arXiv: https://arxiv.org/abs/2412.14166 · HTML: https://mlsanity.com/paper/2412.14166/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 100 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)

## Built on by (most cited first, 21 of 21)

- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos](https://mlsanity.com/paper/2506.09997/) (2025)
- [LoMa: Local Feature Matching Revisited](https://mlsanity.com/paper/2604.04931/) (2026)
- [HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers](https://mlsanity.com/paper/2506.03118/) (2025)
- [Vision as Unified Multimodal Generation](https://mlsanity.com/paper/2607.06560/) (2026)
- [I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners](https://mlsanity.com/paper/2512.13683/) (2025)
- [Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data](https://mlsanity.com/paper/2509.06950/) (2025)
- [2Xplat: Decoupling Geometry and Appearance Modeling for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2603.21064/) (2026)
- [SimpleProc: Fully Procedural Synthetic Data from Simple Rules for Multi-View Stereo](https://mlsanity.com/paper/2604.04925/) (2026)
- [SynFlow: Scaling Up LiDAR Scene Flow Estimation with Synthetic Data](https://mlsanity.com/paper/2604.09411/) (2026)
- [Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?](https://mlsanity.com/paper/2609.34621/) (2026)
- [FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation](https://mlsanity.com/paper/2604.10512/) (2026)
- [Who Handles Orientation? Investigating Invariance in Feature Matching](https://mlsanity.com/paper/2604.11809/) (2026)
- [DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers](https://mlsanity.com/paper/2606.31585/) (2026)
- [Room Envelopes: A Synthetic Dataset for Indoor Layout Reconstruction from Images](https://mlsanity.com/paper/2511.03970/) (2025)
- [Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models](https://mlsanity.com/paper/2604.10095/) (2026)
- [ForeSplat: Optimization-Aware Foresight for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2605.22020/) (2026)
- [RoMa-Omega: What Feed-Forward 3D Models Know About Image Matching](https://mlsanity.com/paper/2609.09507/) (2026)

## Abstract (arXiv)

We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data. Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains. Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction. Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization, as well as application to other tasks.