# Zero-Shot Multi-Object Scene Completion

arXiv: https://arxiv.org/abs/2403.14628 · HTML: https://mlsanity.com/paper/2403.14628/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 80 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 30 of 72)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)

## Built on by (most cited first, 9 of 9)

- [ZeroGrasp: Zero-Shot Shape Reconstruction Enabled Robotic Grasping](https://mlsanity.com/paper/2504.10857/) (2025)
- [LaS-Comp: Zero-shot 3D Completion with Latent-Spatial Consistency](https://mlsanity.com/paper/2602.18735/) (2026)
- [Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling](https://mlsanity.com/paper/2602.08058/) (2026)
- [Affostruction: 3D Affordance Grounding with Generative Reconstruction](https://mlsanity.com/paper/2601.09211/) (2026)
- [GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors](https://mlsanity.com/paper/2606.08440/) (2026)
- [FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction](https://mlsanity.com/paper/2606.21373/) (2026)
- [Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planning](https://mlsanity.com/paper/2411.19461/) (2024)
- [GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity](https://mlsanity.com/paper/2505.11905/) (2025)
- [CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image](https://mlsanity.com/paper/2609.25654/) (2026)

## Abstract (arXiv)

We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Despite notable advancements in single-object 3D shape completion, high-quality reconstructions in highly cluttered real-world multi-object scenes remains a challenge. To address this issue, we propose OctMAE, an architecture that leverages an Octree U-Net and a latent 3D MAE to achieve high-quality and near real-time multi-object scene completion through both local and global geometric reasoning. Because a naive 3D MAE can be computationally intractable and memory intensive even in the latent space, we introduce a novel occlusion masking strategy and adopt 3D rotary embeddings, which significantly improves the runtime and scene completion quality. To generalize to a wide range of objects in diverse scenes, we create a large-scale photorealistic dataset, featuring a diverse set of 12K 3D object models from the Objaverse dataset which are rendered in multi-object scenes with physics-based positioning. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets and demonstrates a strong zero-shot capability.