# FroDO: From Detections to 3D Objects

arXiv: https://arxiv.org/abs/2005.05125 · HTML: https://mlsanity.com/paper/2005.05125/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 51)

- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [CodeNeRF: Disentangled Neural Radiance Fields for Object Categories](https://mlsanity.com/paper/2109.01750/) (2021)
- [SMPLicit: Topology-aware Generative Model for Clothed People](https://mlsanity.com/paper/2103.06871/) (2021)
- [CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation](https://mlsanity.com/paper/2203.01929/) (2022)
- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [OmniGlue: Generalizable Feature Matching with Foundation Model Guidance](https://mlsanity.com/paper/2405.12979/) (2024)
- [Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches](https://mlsanity.com/paper/2104.00482/) (2021)
- [DSP-SLAM: Object Oriented SLAM with Deep Shape Priors](https://mlsanity.com/paper/2108.09481/) (2021)
- [In-Hand 3D Object Scanning from an RGB Sequence](https://mlsanity.com/paper/2211.16193/) (2022)
- [ODAM: Object Detection, Association, and Mapping using Posed RGB Video](https://mlsanity.com/paper/2108.10165/) (2021)
- [Pose2Room: Understanding 3D Scenes from Human Activities](https://mlsanity.com/paper/2112.03030/) (2021)
- [Category-Aware Transformer Network for Better Human-Object Interaction Detection](https://mlsanity.com/paper/2204.04911/) (2022)
- [ShapeR: Robust Conditional 3D Shape Generation from Casual Captures](https://mlsanity.com/paper/2601.11514/) (2026)
- [Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image](https://mlsanity.com/paper/2111.03098/) (2021)
- [PINs: Progressive Implicit Networks for Multi-Scale Neural Representations](https://mlsanity.com/paper/2202.04713/) (2022)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds](https://mlsanity.com/paper/2204.09138/) (2022)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots](https://mlsanity.com/paper/2303.08923/) (2023)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detection](https://mlsanity.com/paper/2310.01401/) (2023)
- [Multi-person Implicit Reconstruction from a Single Image](https://mlsanity.com/paper/2104.09283/) (2021)
- [AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings](https://mlsanity.com/paper/2406.08960/) (2024)
- [ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Description](https://mlsanity.com/paper/2108.00355/) (2021)
- [Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos](https://mlsanity.com/paper/2104.12276/) (2021)
- [A Real World Dataset for Multi-view 3D Reconstruction](https://mlsanity.com/paper/2203.11397/) (2022)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model](https://mlsanity.com/paper/2308.09591/) (2023)
- [Uncertainty-aware 3D Object-Level Mapping with Deep Shape Priors](https://mlsanity.com/paper/2309.09118/) (2023)
- [SDFEst: Categorical Pose and Shape Estimation of Objects from RGB-D using Signed Distance Fields](https://mlsanity.com/paper/2207.04880/) (2022)

## Abstract (arXiv)

Object-oriented maps are important for scene understanding since they jointly capture geometry and semantics, allow individual instantiation and meaningful reasoning about objects. We introduce FroDO, a method for accurate 3D reconstruction of object instances from RGB video that infers object location, pose and shape in a coarse-to-fine manner. Key to FroDO is to embed object shapes in a novel learnt space that allows seamless switching between sparse point cloud and dense DeepSDF decoding. Given an input sequence of localized RGB frames, FroDO first aggregates 2D detections to instantiate a category-aware 3D bounding box per object. A shape code is regressed using an encoder network before optimizing shape and pose further under the learnt shape priors using sparse and dense shape representations. The optimization uses multi-view geometric, photometric and silhouette losses. We evaluate on real-world datasets, including Pix3D, Redwood-OS, and ScanNet, for single-view, multi-view, and multi-object reconstruction.