# Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

arXiv: https://arxiv.org/abs/1804.10975 · HTML: https://mlsanity.com/paper/1804.10975/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2018.

## Built on by (most cited first, 30 of 33)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors](https://mlsanity.com/paper/2209.02485/) (2022)
- [LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction](https://mlsanity.com/paper/2106.12102/) (2021)
- [Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images](https://mlsanity.com/paper/2204.09866/) (2022)
- [Augmenting Implicit Neural Shape Representations with Explicit Deformation Fields](https://mlsanity.com/paper/2108.08931/) (2021)
- [ANISE: Assembly-based Neural Implicit Surface rEconstruction](https://mlsanity.com/paper/2205.13682/) (2022)
- [Optimal Pose and Shape Estimation for Category-level 3D Object Perception](https://mlsanity.com/paper/2104.08383/) (2021)
- [3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2110.08861/) (2021)
- [DeepMesh: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2106.11795/) (2021)
- [Semi-Supervised Single-View 3D Reconstruction via Prototype Shape Priors](https://mlsanity.com/paper/2209.15383/) (2022)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction](https://mlsanity.com/paper/2312.02221/) (2023)
- [Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion](https://mlsanity.com/paper/2507.06230/) (2025)
- [Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videos](https://mlsanity.com/paper/2111.14465/) (2021)
- [Deep Generative Model for Efficient 3D Airfoil Parameterization and Generation](https://mlsanity.com/paper/2101.02744/) (2021)
- [Single-view 3D Mesh Reconstruction for Seen and Unseen Categories](https://mlsanity.com/paper/2208.02676/) (2022)
- [Fostering Generalization in Single-view 3D Reconstruction by Learning a Hierarchy of Local and Global Shape Priors](https://mlsanity.com/paper/2104.00476/) (2021)
- [Few-shot Single-view 3D Reconstruction with Memory Prior Contrastive Network](https://mlsanity.com/paper/2208.00183/) (2022)
- [GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff](https://mlsanity.com/paper/2211.02299/) (2022)
- [Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints](https://mlsanity.com/paper/2206.12498/) (2022)
- [Multi-Category Mesh Reconstruction From Image Collections](https://mlsanity.com/paper/2110.11256/) (2021)
- [SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation](https://mlsanity.com/paper/2509.12721/) (2025)
- [AE-NeRF: Auto-Encoding Neural Radiance Fields for 3D-Aware Object Manipulation](https://mlsanity.com/paper/2204.13426/) (2022)
- [A Review of Deep Learning-Powered Mesh Reconstruction Methods](https://mlsanity.com/paper/2303.02879/) (2023)
- [Learning Compositional Shape Priors for Few-Shot 3D Reconstruction](https://mlsanity.com/paper/2106.06440/) (2021)
- [See360: Novel Panoramic View Interpolation](https://mlsanity.com/paper/2401.03431/) (2024)
- [DDR-Net: Learning Multi-Stage Multi-View Stereo With Dynamic Depth Range](https://mlsanity.com/paper/2103.14275/) (2021)
- [HyperColor: A HyperNetwork Approach for Synthesizing Auto-colored 3D Models for Game Scenes Population](https://mlsanity.com/paper/2108.01411/) (2021)
- [Domain Adaptation for Real-World Single View 3D Reconstruction](https://mlsanity.com/paper/2108.10972/) (2021)
- [PT43D: A Probabilistic Transformer for Generating 3D Shapes from Single Highly-Ambiguous RGB Images](https://mlsanity.com/paper/2405.11914/) (2024)

## Abstract (arXiv)

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.