# DDM-NET: End-to-end learning of keypoint feature Detection, Description and Matching for 3D localization

arXiv: https://arxiv.org/abs/2212.04575 · HTML: https://mlsanity.com/paper/2212.04575/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 28 of 28)

- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Neighbourhood Consensus Networks](https://mlsanity.com/paper/1810.10510/) (2018)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [Patch2Pix: Epipolar-Guided Pixel-Level Correspondences](https://mlsanity.com/paper/2012.01909/) (2020)
- [Learning Feature Descriptors using Camera Pose Supervision](https://mlsanity.com/paper/2004.13324/) (2020)
- [Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions](https://mlsanity.com/paper/2004.10566/) (2020)
- [Dual-Resolution Correspondence Networks](https://mlsanity.com/paper/2006.08844/) (2020)
- [ContextDesc: Local Descriptor Augmentation with Cross-Modality Context](https://mlsanity.com/paper/1904.04084/) (2019)
- [GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints](https://mlsanity.com/paper/1807.06294/) (2018)
- [GIFT: Learning Transformation-Invariant Dense Visual Descriptors via Group CNNs](https://mlsanity.com/paper/1911.05932/) (2019)
- [D2D: Keypoint Extraction with Describe to Detect Approach](https://mlsanity.com/paper/2005.13605/) (2020)
- [Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints](https://mlsanity.com/paper/2007.15122/) (2020)
- [Decoupling Makes Weakly Supervised Local Feature Better](https://mlsanity.com/paper/2201.02861/) (2022)
- [Self-supervised Geometric Perception](https://mlsanity.com/paper/2103.03114/) (2021)
- [UnsuperPoint: End-to-end Unsupervised Interest Point Detector and Descriptor](https://mlsanity.com/paper/1907.04011/) (2019)
- [Deep learning and face recognition: the state of the art](https://mlsanity.com/paper/1902.03524/) (2019)
- [Same Features, Different Day: Weakly Supervised Feature Learning for Seasonal Invariance](https://mlsanity.com/paper/2003.13431/) (2020)

## Abstract (arXiv)

In this paper, we propose an end-to-end framework that jointly learns keypoint detection, descriptor representation and cross-frame matching for the task of image-based 3D localization. Prior art has tackled each of these components individually, purportedly aiming to alleviate difficulties in effectively train a holistic network. We design a self-supervised image warping correspondence loss for both feature detection and matching, a weakly-supervised epipolar constraints loss on relative camera pose learning, and a directional matching scheme that detects key-point features in a source image and performs coarse-to-fine correspondence search on the target image. We leverage this framework to enforce cycle consistency in our matching module. In addition, we propose a new loss to robustly handle both definite inlier/outlier matches and less-certain matches. The integration of these learning mechanisms enables end-to-end training of a single network performing all three localization components. Bench-marking our approach on public data-sets, exemplifies how such an end-to-end framework is able to yield more accurate localization that out-performs both traditional methods as well as state-of-the-art weakly supervised methods.