# End-to-end Learning the Partial Permutation Matrix for Robust 3D Point Cloud Registration

arXiv: https://arxiv.org/abs/2110.15250 · HTML: https://mlsanity.com/paper/2110.15250/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020)
- [PointNetLK: Robust & Efficient Point Cloud Registration using PointNet](https://mlsanity.com/paper/1903.05711/) (2019)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [PPFNet: Global Context Aware Local Features for Robust 3D Point Matching](https://mlsanity.com/paper/1802.02669/) (2018)
- [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration](https://mlsanity.com/paper/1605.03344/) (2016)
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](https://mlsanity.com/paper/1910.12240/) (2019)
- [PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency](https://mlsanity.com/paper/2103.05465/) (2021)
- [PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors](https://mlsanity.com/paper/1808.10322/) (2018)
- [Feature-metric Registration: A Fast Semi-supervised Approach for Robust Point Cloud Registration without Correspondences](https://mlsanity.com/paper/2005.01014/) (2020)
- [3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration](https://mlsanity.com/paper/1807.09413/) (2018)
- [3DRegNet: A Deep Neural Network for 3D Point Registration](https://mlsanity.com/paper/1904.01701/) (2019)
- [DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration](https://mlsanity.com/paper/1905.04153/) (2019)
- [CorrNet3D: Unsupervised End-to-end Learning of Dense Correspondence for 3D Point Clouds](https://mlsanity.com/paper/2012.15638/) (2020)
- [Quaternion Equivariant Capsule Networks for 3D Point Clouds](https://mlsanity.com/paper/1912.12098/) (2019)
- [IMLS-SLAM: scan-to-model matching based on 3D data](https://mlsanity.com/paper/1802.08633/) (2018)
- [SegICP: Integrated Deep Semantic Segmentation and Pose Estimation](https://mlsanity.com/paper/1703.01661/) (2017)
- [DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds](https://mlsanity.com/paper/1811.11397/) (2018)
- [One Framework to Register Them All: PointNet Encoding for Point Cloud Alignment](https://mlsanity.com/paper/1912.05766/) (2019)

## Built on by (most cited first, 5 of 5)

- [GeoTransformer: Fast and Robust Point Cloud Registration with Geometric Transformer](https://mlsanity.com/paper/2308.03768/) (2023)
- [VRNet: Learning the Rectified Virtual Corresponding Points for 3D Point Cloud Registration](https://mlsanity.com/paper/2203.13241/) (2022)
- [UTOPIC: Uncertainty-aware Overlap Prediction Network for Partial Point Cloud Registration](https://mlsanity.com/paper/2208.02712/) (2022)
- [DBDNet:Partial-to-Partial Point Cloud Registration with Dual Branches Decoupling](https://mlsanity.com/paper/2310.11733/) (2023)
- [3D Registration in 30 Years: A Survey](https://mlsanity.com/paper/2412.13735/) (2024)

## Abstract (arXiv)

Even though considerable progress has been made in deep learning-based 3D point cloud processing, how to obtain accurate correspondences for robust registration remains a major challenge because existing hard assignment methods cannot deal with outliers naturally. Alternatively, the soft matching-based methods have been proposed to learn the matching probability rather than hard assignment. However, in this paper, we prove that these methods have an inherent ambiguity causing many deceptive correspondences. To address the above challenges, we propose to learn a partial permutation matching matrix, which does not assign corresponding points to outliers, and implements hard assignment to prevent ambiguity. However, this proposal poses two new problems, i.e., existing hard assignment algorithms can only solve a full rank permutation matrix rather than a partial permutation matrix, and this desired matrix is defined in the discrete space, which is non-differentiable. In response, we design a dedicated soft-to-hard (S2H) matching procedure within the registration pipeline consisting of two steps: solving the soft matching matrix (S-step) and projecting this soft matrix to the partial permutation matrix (H-step). Specifically, we augment the profit matrix before the hard assignment to solve an augmented permutation matrix, which is cropped to achieve the final partial permutation matrix. Moreover, to guarantee end-to-end learning, we supervise the learned partial permutation matrix but propagate the gradient to the soft matrix instead. Our S2H matching procedure can be easily integrated with existing registration frameworks, which has been verified in representative frameworks including DCP, RPMNet, and DGR. Extensive experiments have validated our method, which creates a new state-of-the-art performance for robust 3D point cloud registration. The code will be made public.