# TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation

arXiv: https://arxiv.org/abs/2303.13477 · HTML: https://mlsanity.com/paper/2303.13477/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation](https://mlsanity.com/paper/2007.08454/) (2020)
- [Solving ill-posed inverse problems using iterative deep neural networks](https://mlsanity.com/paper/1704.04058/) (2017)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanism](https://mlsanity.com/paper/2103.07054/) (2021)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [Category-Level 6D Object Pose and Size Estimation using Self-Supervised Deep Prior Deformation Networks](https://mlsanity.com/paper/2207.05444/) (2022)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction](https://mlsanity.com/paper/2004.04485/) (2020)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos](https://mlsanity.com/paper/2012.04641/) (2020)
- [GN-Net: The Gauss-Newton Loss for Multi-Weather Relocalization](https://mlsanity.com/paper/1904.11932/) (2019)
- [DSP-SLAM: Object Oriented SLAM with Deep Shape Priors](https://mlsanity.com/paper/2108.09481/) (2021)
- [Inverse Compositional Spatial Transformer Networks](https://mlsanity.com/paper/1612.03897/) (2016)
- [LS-Net: Learning to Solve Nonlinear Least Squares for Monocular Stereo](https://mlsanity.com/paper/1809.02966/) (2018)
- [EAO-SLAM: Monocular Semi-Dense Object SLAM Based on Ensemble Data Association](https://mlsanity.com/paper/2004.12730/) (2020)
- [iCaps: Iterative Category-level Object Pose and Shape Estimation](https://mlsanity.com/paper/2201.00059/) (2021)
- [Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation](https://mlsanity.com/paper/2203.14531/) (2022)
- [Representing 3D Shapes with Probabilistic Directed Distance Fields](https://mlsanity.com/paper/2112.05300/) (2021)

## Built on by (most cited first, 2 of 2)

- [DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field](https://mlsanity.com/paper/2308.08231/) (2023)
- [Neural directional distance field object representation for uni-directional path-traced rendering](https://mlsanity.com/paper/2306.16142/) (2023)

## Abstract (arXiv)

We propose a novel method for joint estimation of shape and pose of rigid objects from their sequentially observed RGB-D images. In sharp contrast to past approaches that rely on complex non-linear optimization, we propose to formulate it as a neural optimization that learns to efficiently estimate the shape and pose. We introduce Deep Directional Distance Function (DeepDDF), a neural network that directly outputs the depth image of an object given the camera viewpoint and viewing direction, for efficient error computation in 2D image space. We formulate the joint estimation itself as a Transformer which we refer to as TransPoser. We fully leverage the tokenization and multi-head attention to sequentially process the growing set of observations and to efficiently update the shape and pose with a learned momentum, respectively. Experimental results on synthetic and real data show that DeepDDF achieves high accuracy as a category-level object shape representation and TransPoser achieves state-of-the-art accuracy efficiently for joint shape and pose estimation.