# Learning Correspondence Uncertainty via Differentiable Nonlinear Least Squares

arXiv: https://arxiv.org/abs/2305.09527 · HTML: https://mlsanity.com/paper/2305.09527/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 77 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry](https://mlsanity.com/paper/1807.02570/) (2018)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [Visual-Inertial Mapping with Non-Linear Factor Recovery](https://mlsanity.com/paper/1904.06504/) (2019)
- [GN-Net: The Gauss-Newton Loss for Multi-Weather Relocalization](https://mlsanity.com/paper/1904.11932/) (2019)
- [gradSLAM: Automagically differentiable SLAM](https://mlsanity.com/paper/1910.10672/) (2019)
- [S2DNet: Learning Accurate Correspondences for Sparse-to-Dense Feature Matching](https://mlsanity.com/paper/2004.01673/) (2020)
- [Geometric Consistency for Self-Supervised End-to-End Visual Odometry](https://mlsanity.com/paper/1804.03789/) (2018)
- [Radar Odometry Combining Probabilistic Estimation and Unsupervised Feature Learning](https://mlsanity.com/paper/2105.14152/) (2021)
- [The Probabilistic Normal Epipolar Constraint for Frame-To-Frame Rotation Optimization under Uncertain Feature Positions](https://mlsanity.com/paper/2204.02256/) (2022)
- [Monocular Rotational Odometry with Incremental Rotation Averaging and Loop Closure](https://mlsanity.com/paper/2010.01872/) (2020)
- [Quaternion Based Camera Pose Estimation From Matched Feature Points](https://mlsanity.com/paper/1704.02672/) (2017)

## Built on by (most cited first, 14 of 14)

- [Robust Incremental Structure-from-Motion with Hybrid Features](https://mlsanity.com/paper/2409.19811/) (2024)
- [MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry](https://mlsanity.com/paper/2409.09479/) (2024)
- [AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos](https://mlsanity.com/paper/2503.23282/) (2025)
- [MambaVO: Deep Visual Odometry Based on Sequential Matching Refinement and Training Smoothing](https://mlsanity.com/paper/2412.20082/) (2024)
- [DAC: Detector-Agnostic Spatial Covariances for Deep Local Features](https://mlsanity.com/paper/2305.12250/) (2023)
- [RaCo: Ranking and Covariance for Practical Learned Keypoints](https://mlsanity.com/paper/2602.15755/) (2026)
- [BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction](https://mlsanity.com/paper/2503.19340/) (2025)
- [From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers](https://mlsanity.com/paper/2406.07785/) (2024)
- [ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching](https://mlsanity.com/paper/2604.03377/) (2026)
- [A Pose-only Geometric Constraint for Multi-Camera Pose Adjustment](https://mlsanity.com/paper/2604.23704/) (2026)
- [VOCA: Visual Odometry with Codec Awareness](https://mlsanity.com/paper/2607.00189/) (2026)
- [Residual Learning for Image Point Descriptors](https://mlsanity.com/paper/2312.15471/) (2023)
- [KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry](https://mlsanity.com/paper/2608.24544/) (2026)
- [MomentBA: Second-order Spatial Moments for Anisotropic Correspondence Uncertainty in Differentiable Bundle Adjustment](https://mlsanity.com/paper/2609.13691/) (2026)

## Abstract (arXiv)

We propose a differentiable nonlinear least squares framework to account for uncertainty in relative pose estimation from feature correspondences. Specifically, we introduce a symmetric version of the probabilistic normal epipolar constraint, and an approach to estimate the covariance of feature positions by differentiating through the camera pose estimation procedure. We evaluate our approach on synthetic, as well as the KITTI and EuRoC real-world datasets. On the synthetic dataset, we confirm that our learned covariances accurately approximate the true noise distribution. In real world experiments, we find that our approach consistently outperforms state-of-the-art non-probabilistic and probabilistic approaches, regardless of the feature extraction algorithm of choice.