# SDFEst: Categorical Pose and Shape Estimation of Objects from RGB-D using Signed Distance Fields

arXiv: https://arxiv.org/abs/2207.04880 · HTML: https://mlsanity.com/paper/2207.04880/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation](https://mlsanity.com/paper/2007.08454/) (2020)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)
- [Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/2001.09322/) (2020)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Category Level Object Pose Estimation via Neural Analysis-by-Synthesis](https://mlsanity.com/paper/2008.08145/) (2020)
- [Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era](https://mlsanity.com/paper/1906.06543/) (2019)
- [Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images](https://mlsanity.com/paper/2006.12250/) (2020)
- [CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation](https://mlsanity.com/paper/2203.01929/) (2022)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [A Large Dataset of Object Scans](https://mlsanity.com/paper/1602.02481/) (2016)

## Built on by (most cited first, 7 of 7)

- [SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations](https://mlsanity.com/paper/2303.10346/) (2023)
- [OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation](https://mlsanity.com/paper/2403.12396/) (2024)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [RGB-D-Based Categorical Object Pose and Shape Estimation: Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2301.08147/) (2023)
- [TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation](https://mlsanity.com/paper/2303.13477/) (2023)
- [ShapeICP: Iterative Category-level Object Pose and Shape Estimation from Depth](https://mlsanity.com/paper/2408.13147/) (2024)
- [Learning Positive-Incentive Point Sampling in Neural Implicit Fields for Object Pose Estimation](https://mlsanity.com/paper/2602.19937/) (2026)

## Abstract (arXiv)

Rich geometric understanding of the world is an important component of many robotic applications such as planning and manipulation. In this paper, we present a modular pipeline for pose and shape estimation of objects from RGB-D images given their category. The core of our method is a generative shape model, which we integrate with a novel initialization network and a differentiable renderer to enable 6D pose and shape estimation from a single or multiple views. We investigate the use of discretized signed distance fields as an efficient shape representation for fast analysis-by-synthesis optimization. Our modular framework enables multi-view optimization and extensibility. We demonstrate the benefits of our approach over state-of-the-art methods in several experiments on both synthetic and real data. We open-source our approach at https://github.com/roym899/sdfest.