# MarrNet: 3D Shape Reconstruction via 2.5D Sketches

arXiv: https://arxiv.org/abs/1711.03129 · HTML: https://mlsanity.com/paper/1711.03129/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2017.

## Built on by (most cited first, 30 of 108)

- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2312.09147/) (2023)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views](https://mlsanity.com/paper/2211.16431/) (2022)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022)
- [Multiview Compressive Coding for 3D Reconstruction](https://mlsanity.com/paper/2301.08247/) (2023)
- [3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models](https://mlsanity.com/paper/2212.00842/) (2022)
- [Neural Fields as Learnable Kernels for 3D Reconstruction](https://mlsanity.com/paper/2111.13674/) (2021)
- [GIFS: Neural Implicit Function for General Shape Representation](https://mlsanity.com/paper/2204.07126/) (2022)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution](https://mlsanity.com/paper/2103.14230/) (2021)
- [DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model](https://mlsanity.com/paper/2304.02827/) (2023)
- [SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images](https://mlsanity.com/paper/2501.04689/) (2025)
- [SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data](https://mlsanity.com/paper/2105.08612/) (2021)
- [NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields](https://mlsanity.com/paper/2203.01762/) (2022)
- [Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes](https://mlsanity.com/paper/2206.04942/) (2022)
- [Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion](https://mlsanity.com/paper/2211.11674/) (2022)
- [ZeroShape: Regression-based Zero-shot Shape Reconstruction](https://mlsanity.com/paper/2312.14198/) (2023)
- [Topologically-Aware Deformation Fields for Single-View 3D Reconstruction](https://mlsanity.com/paper/2205.06267/) (2022)
- [CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wild](https://mlsanity.com/paper/2203.03089/) (2022)
- [PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds](https://mlsanity.com/paper/2109.05566/) (2021)
- [SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators](https://mlsanity.com/paper/2201.00112/) (2022)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)

## Abstract (arXiv)

3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.