# Real3D: Scaling Up Large Reconstruction Models with Real-World Images

arXiv: https://arxiv.org/abs/2406.08479 · HTML: https://mlsanity.com/paper/2406.08479/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 96 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 30 of 81)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)

## Built on by (most cited first, 24 of 24)

- [3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion](https://mlsanity.com/paper/2409.12957/) (2024)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement](https://mlsanity.com/paper/2408.00653/) (2024)
- [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025)
- [LRM-Zero: Training Large Reconstruction Models with Synthesized Data](https://mlsanity.com/paper/2406.09371/) (2024)
- [MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data](https://mlsanity.com/paper/2412.14166/) (2024)
- [LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models](https://mlsanity.com/paper/2412.07746/) (2024)
- [4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time](https://mlsanity.com/paper/2506.18890/) (2025)
- [Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos](https://mlsanity.com/paper/2507.12646/) (2025)
- [Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation](https://mlsanity.com/paper/2410.00890/) (2024)
- [Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles](https://mlsanity.com/paper/2412.14494/) (2024)
- [GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction](https://mlsanity.com/paper/2410.13911/) (2024)
- [Dust to Tower: Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images](https://mlsanity.com/paper/2412.19518/) (2024)
- [3D Arena: An Open Platform for Generative 3D Evaluation](https://mlsanity.com/paper/2506.18787/) (2025)
- [RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image](https://mlsanity.com/paper/2410.08181/) (2024)
- [LabelAny3D: Label Any Object 3D in the Wild](https://mlsanity.com/paper/2601.01676/) (2026)
- [Instructive3D: Editing Large Reconstruction Models with Text Instructions](https://mlsanity.com/paper/2501.04374/) (2025)
- [Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors](https://mlsanity.com/paper/2603.18782/) (2026)
- [F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Aggregative Gaussian Splatting](https://mlsanity.com/paper/2501.06714/) (2025)
- [Fast Spatial Memory with Elastic Test-Time Training](https://mlsanity.com/paper/2604.07350/) (2026)
- [GeoFusionLRM: Geometry-Aware Self-Correction for Consistent 3D Reconstruction](https://mlsanity.com/paper/2602.14119/) (2026)
- [HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation](https://mlsanity.com/paper/2511.00468/) (2025)
- [HumanNOVA: Photorealistic, Universal and Rapid 3D Human Avatar Modeling from a Single Image](https://mlsanity.com/paper/2606.02573/) (2026)
- [Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation](https://mlsanity.com/paper/2608.23206/) (2026)

## Abstract (arXiv)

The default strategy for training single-view Large Reconstruction Models (LRMs) follows the fully supervised route using large-scale datasets of synthetic 3D assets or multi-view captures. Although these resources simplify the training procedure, they are hard to scale up beyond the existing datasets and they are not necessarily representative of the real distribution of object shapes. To address these limitations, in this paper, we introduce Real3D, the first LRM system that can be trained using single-view real-world images. Real3D introduces a novel self-training framework that can benefit from both the existing synthetic data and diverse single-view real images. We propose two unsupervised losses that allow us to supervise LRMs at the pixel- and semantic-level, even for training examples without ground-truth 3D or novel views. To further improve performance and scale up the image data, we develop an automatic data curation approach to collect high-quality examples from in-the-wild images. Our experiments show that Real3D consistently outperforms prior work in four diverse evaluation settings that include real and synthetic data, as well as both in-domain and out-of-domain shapes. Code and model can be found here: https://hwjiang1510.github.io/Real3D/