# Hierarchical Metric Learning and Matching for 2D and 3D Geometric Correspondences

arXiv: https://arxiv.org/abs/1803.07231 · HTML: https://mlsanity.com/paper/1803.07231/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2018.

## Built on by (most cited first, 11 of 11)

- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective](https://mlsanity.com/paper/2203.10887/) (2022)
- [Learning Hierarchical Graph Neural Networks for Image Clustering](https://mlsanity.com/paper/2107.01319/) (2021)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [ZippyPoint: Fast Interest Point Detection, Description, and Matching through Mixed Precision Discretization](https://mlsanity.com/paper/2203.03610/) (2022)
- [The Change You Want to See](https://mlsanity.com/paper/2209.14341/) (2022)
- [LoCUS: Learning Multiscale 3D-consistent Features from Posed Images](https://mlsanity.com/paper/2310.01095/) (2023)
- [FPC-Net: Revisiting SuperPoint with Descriptor-Free Keypoint Detection via Feature Pyramids and Consistency-Based Implicit Matching](https://mlsanity.com/paper/2507.10770/) (2025)
- [Domain Generalized Stereo Matching with Uncertainty-guided Data Augmentation](https://mlsanity.com/paper/2508.01303/) (2025)

## Abstract (arXiv)

Interest point descriptors have fueled progress on almost every problem in computer vision. Recent advances in deep neural networks have enabled task-specific learned descriptors that outperform hand-crafted descriptors on many problems. We demonstrate that commonly used metric learning approaches do not optimally leverage the feature hierarchies learned in a Convolutional Neural Network (CNN), especially when applied to the task of geometric feature matching. While a metric loss applied to the deepest layer of a CNN, is often expected to yield ideal features irrespective of the task, in fact the growing receptive field as well as striding effects cause shallower features to be better at high precision matching tasks. We leverage this insight together with explicit supervision at multiple levels of the feature hierarchy for better regularization, to learn more effective descriptors in the context of geometric matching tasks. Further, we propose to use activation maps at different layers of a CNN, as an effective and principled replacement for the multi-resolution image pyramids often used for matching tasks. We propose concrete CNN architectures employing these ideas, and evaluate them on multiple datasets for 2D and 3D geometric matching as well as optical flow, demonstrating state-of-the-art results and generalization across datasets.