# End-to-end learning of keypoint detector and descriptor for pose invariant 3D matching

arXiv: https://arxiv.org/abs/1802.07869 · HTML: https://mlsanity.com/paper/1802.07869/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 4 of 4)

- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)
- [SpineOne: A One-Stage Detection Framework for Degenerative Discs and Vertebrae](https://mlsanity.com/paper/2110.15082/) (2021)
- [Review on 6D Object Pose Estimation with the focus on Indoor Scene Understanding](https://mlsanity.com/paper/2212.01920/) (2022)
- [EOPose : Exemplar-based object reposing using Generalized Pose Correspondences](https://mlsanity.com/paper/2505.03394/) (2025)

## Abstract (arXiv)

Finding correspondences between images or 3D scans is at the heart of many computer vision and image retrieval applications and is often enabled by matching local keypoint descriptors. Various learning approaches have been applied in the past to different stages of the matching pipeline, considering detector, descriptor, or metric learning objectives. These objectives were typically addressed separately and most previous work has focused on image data. This paper proposes an end-to-end learning framework for keypoint detection and its representation (descriptor) for 3D depth maps or 3D scans, where the two can be jointly optimized towards task-specific objectives without a need for separate annotations. We employ a Siamese architecture augmented by a sampling layer and a novel score loss function which in turn affects the selection of region proposals. The positive and negative examples are obtained automatically by sampling corresponding region proposals based on their consistency with known 3D pose labels. Matching experiments with depth data on multiple benchmark datasets demonstrate the efficacy of the proposed approach, showing significant improvements over state-of-the-art methods.