# Neural Outlier Rejection for Self-Supervised Keypoint Learning

arXiv: https://arxiv.org/abs/1912.10615 · HTML: https://mlsanity.com/paper/1912.10615/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 16 of 16)

- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [SiLK -- Simple Learned Keypoints](https://mlsanity.com/paper/2304.06194/) (2023)
- [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024)
- [ZippyPoint: Fast Interest Point Detection, Description, and Matching through Mixed Precision Discretization](https://mlsanity.com/paper/2203.03610/) (2022)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [DeLiRa: Self-Supervised Depth, Light, and Radiance Fields](https://mlsanity.com/paper/2304.02797/) (2023)
- [Digging Into Self-Supervised Learning of Feature Descriptors](https://mlsanity.com/paper/2110.04773/) (2021)
- [Consensus Learning with Deep Sets for Essential Matrix Estimation](https://mlsanity.com/paper/2406.17414/) (2024)
- [EdgePoint2: Compact Descriptors for Superior Efficiency and Accuracy](https://mlsanity.com/paper/2504.17280/) (2025)
- [Efficient Feature Description for Small Body Relative Navigation using Binary Convolutional Neural Networks](https://mlsanity.com/paper/2304.04985/) (2023)
- [Long-Term Invariant Local Features via Implicit Cross-Domain Correspondences](https://mlsanity.com/paper/2311.03345/) (2023)
- [Residual Learning for Image Point Descriptors](https://mlsanity.com/paper/2312.15471/) (2023)
- [CLIDD: Cross-Layer Independent Deformable Description for Efficient and Discriminative Local Feature Representation](https://mlsanity.com/paper/2601.09230/) (2026)
- [XRefine: Attention-Guided Keypoint Match Refinement](https://mlsanity.com/paper/2601.12530/) (2026)

## Abstract (arXiv)

Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks. However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators. We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching. Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description. We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution. Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.