# Learning Geodesic-Aware Local Features from RGB-D Images

arXiv: https://arxiv.org/abs/2203.12016 · HTML: https://mlsanity.com/paper/2203.12016/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 14 of 14)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Image Matching across Wide Baselines: From Paper to Practice](https://mlsanity.com/paper/2003.01587/) (2020)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters](https://mlsanity.com/paper/1904.00889/) (2019)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [Detect-to-Retrieve: Efficient Regional Aggregation for Image Search](https://mlsanity.com/paper/1812.01584/) (2018)
- [Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascade](https://mlsanity.com/paper/1810.12163/) (2018)
- [Beyond Cartesian Representations for Local Descriptors](https://mlsanity.com/paper/1908.05547/) (2019)
- [Incremental Class Discovery for Semantic Segmentation with RGBD Sensing](https://mlsanity.com/paper/1907.10008/) (2019)
- [Geodesic Distance Descriptors](https://mlsanity.com/paper/1611.07360/) (2016)

## Built on by (most cited first, 5 of 5)

- [XFeat: Accelerated Features for Lightweight Image Matching](https://mlsanity.com/paper/2404.19174/) (2024)
- [Enhancing Deformable Local Features by Jointly Learning to Detect and Describe Keypoints](https://mlsanity.com/paper/2304.00583/) (2023)
- [Leveraging Semantic Cues from Foundation Vision Models for Enhanced Local Feature Correspondence](https://mlsanity.com/paper/2410.09533/) (2024)
- [Learning to Detect Good Keypoints to Match Non-Rigid Objects in RGB Images](https://mlsanity.com/paper/2212.09589/) (2022)
- [Improving the matching of deformable objects by learning to detect keypoints](https://mlsanity.com/paper/2309.00434/) (2023)

## Abstract (arXiv)

Most of the existing handcrafted and learning-based local descriptors are still at best approximately invariant to affine image transformations, often disregarding deformable surfaces. In this paper, we take one step further by proposing a new approach to compute descriptors from RGB-D images (where RGB refers to the pixel color brightness and D stands for depth information) that are invariant to isometric non-rigid deformations, as well as to scale changes and rotation. Our proposed description strategies are grounded on the key idea of learning feature representations on undistorted local image patches using surface geodesics. We design two complementary local descriptors strategies to compute geodesic-aware features efficiently: one efficient binary descriptor based on handcrafted binary tests (named GeoBit), and one learning-based descriptor (GeoPatch) with convolutional neural networks (CNNs) to compute features. In different experiments using real and publicly available RGB-D data benchmarks, they consistently outperforms state-of-the-art handcrafted and learning-based image and RGB-D descriptors in matching scores, as well as in object retrieval and non-rigid surface tracking experiments, with comparable processing times. We also provide to the community a new dataset with accurate matching annotations of RGB-D images of different objects (shirts, cloths, paintings, bags), subjected to strong non-rigid deformations, for evaluation benchmark of deformable surface correspondence algorithms.