# Canonical mapping as a general-purpose object descriptor for robotic manipulation

arXiv: https://arxiv.org/abs/2303.01331 · HTML: https://mlsanity.com/paper/2303.01331/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [Category Level Object Pose Estimation via Neural Analysis-by-Synthesis](https://mlsanity.com/paper/2008.08145/) (2020)
- [SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddings](https://mlsanity.com/paper/2111.13489/) (2021)
- [Articulation-aware Canonical Surface Mapping](https://mlsanity.com/paper/2004.00614/) (2020)
- [Continuous Surface Embeddings](https://mlsanity.com/paper/2011.12438/) (2020)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Canonical Surface Mapping via Geometric Cycle Consistency](https://mlsanity.com/paper/1907.10043/) (2019)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [Deep Learning of Local RGB-D Patches for 3D Object Detection and 6D Pose Estimation](https://mlsanity.com/paper/1607.06038/) (2016)
- [Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images](https://mlsanity.com/paper/1508.04546/) (2015)
- [GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision](https://mlsanity.com/paper/2007.13124/) (2020)

## Abstract (arXiv)

Perception is an essential part of robotic manipulation in a semi-structured environment. Traditional approaches produce a narrow task-specific prediction (e.g., object's 6D pose), that cannot be adapted to other tasks and is ill-suited for deformable objects. In this paper, we propose using canonical mapping as a near-universal and flexible object descriptor. We demonstrate that common object representations can be derived from a single pre-trained canonical mapping model, which in turn can be generated with minimal manual effort using an automated data generation and training pipeline. We perform a multi-stage experiment using two robot arms that demonstrate the robustness of the perception approach and the ways it can inform the manipulation strategy, thus serving as a powerful foundation for general-purpose robotic manipulation.