# A Model for Multi-View Residual Covariances based on Perspective Deformation

arXiv: https://arxiv.org/abs/2202.00765 · HTML: https://mlsanity.com/paper/2202.00765/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 11 of 11)

- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation](https://mlsanity.com/paper/1911.11789/) (2019)
- [A Photometrically Calibrated Benchmark For Monocular Visual Odometry](https://mlsanity.com/paper/1607.02555/) (2016)
- [Calibration Wizard: A Guidance System for Camera Calibration Based on Modelling Geometric and Corner Uncertainty](https://mlsanity.com/paper/1811.03264/) (2018)
- [Redesigning SLAM for Arbitrary Multi-Camera Systems](https://mlsanity.com/paper/2003.02014/) (2020)
- [Challenges in Monocular Visual Odometry: Photometric Calibration, Motion Bias and Rolling Shutter Effect](https://mlsanity.com/paper/1705.04300/) (2017)
- [Good Feature Matching: Towards Accurate, Robust VO/VSLAM with Low Latency](https://mlsanity.com/paper/2001.00714/) (2020)
- [RGBDTAM: A Cost-Effective and Accurate RGB-D Tracking and Mapping System](https://mlsanity.com/paper/1703.00754/) (2017)
- [Good Feature Selection for Least Squares Pose Optimization in VO/VSLAM](https://mlsanity.com/paper/1905.07807/) (2019)

## Built on by (most cited first, 3 of 3)

- [DAC: Detector-Agnostic Spatial Covariances for Deep Local Features](https://mlsanity.com/paper/2305.12250/) (2023)
- [Single-Shot Metric Depth from Focused Plenoptic Cameras](https://mlsanity.com/paper/2412.02386/) (2024)
- [Motion-Bias-Free Feature-Based SLAM](https://mlsanity.com/paper/2309.06792/) (2023)

## Abstract (arXiv)

In this work, we derive a model for the covariance of the visual residuals in multi-view SfM, odometry and SLAM setups. The core of our approach is the formulation of the residual covariances as a combination of geometric and photometric noise sources. And our key novel contribution is the derivation of a term modelling how local 2D patches suffer from perspective deformation when imaging 3D surfaces around a point. Together, these add up to an efficient and general formulation which not only improves the accuracy of both feature-based and direct methods, but can also be used to estimate more accurate measures of the state entropy and hence better founded point visibility thresholds. We validate our model with synthetic and real data and integrate it into photometric and feature-based Bundle Adjustment, improving their accuracy with a negligible overhead.