# MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry

arXiv: https://arxiv.org/abs/2409.09479 · HTML: https://mlsanity.com/paper/2409.09479/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning](https://mlsanity.com/paper/1709.06841/) (2017)
- [LDSO: Direct Sparse Odometry with Loop Closure](https://mlsanity.com/paper/1808.01111/) (2018)
- [Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras](https://mlsanity.com/paper/1708.07878/) (2017)
- [PyPose: A Library for Robot Learning with Physics-based Optimization](https://mlsanity.com/paper/2209.15428/) (2022)
- [Multivariate Uncertainty in Deep Learning](https://mlsanity.com/paper/1910.14215/) (2019)
- [DiffPoseNet: Direct Differentiable Camera Pose Estimation](https://mlsanity.com/paper/2203.11174/) (2022)
- [Super Odometry: IMU-centric LiDAR-Visual-Inertial Estimator for Challenging Environments](https://mlsanity.com/paper/2104.14938/) (2021)
- [ProbFlow: Joint Optical Flow and Uncertainty Estimation](https://mlsanity.com/paper/1708.06509/) (2017)
- [AirDOS: Dynamic SLAM benefits from Articulated Objects](https://mlsanity.com/paper/2109.09903/) (2021)
- [Learning a Depth Covariance Function](https://mlsanity.com/paper/2303.12157/) (2023)
- [iSLAM: Imperative SLAM](https://mlsanity.com/paper/2306.07894/) (2023)
- [Learning Correspondence Uncertainty via Differentiable Nonlinear Least Squares](https://mlsanity.com/paper/2305.09527/) (2023)

## Built on by (most cited first, 19 of 19)

- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [UFM: A Simple Path towards Unified Dense Correspondence with Flow](https://mlsanity.com/paper/2506.09278/) (2025)
- [Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers](https://mlsanity.com/paper/2511.14751/) (2025)
- [DINO-VO: A Feature-based Visual Odometry Leveraging a Visual Foundation Model](https://mlsanity.com/paper/2507.13145/) (2025)
- [AnyThermal: Towards Learning Universal Representations for Thermal Perception](https://mlsanity.com/paper/2602.06203/) (2026)
- [Keyframe-Based Feed-Forward Visual Odometry](https://mlsanity.com/paper/2601.16020/) (2026)
- [MCGS-SLAM: A Multi-Camera SLAM Framework Using Gaussian Splatting for High-Fidelity Mapping](https://mlsanity.com/paper/2509.14191/) (2025)
- [CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding](https://mlsanity.com/paper/2510.22119/) (2025)
- [MVOFormer: Flow-Semantic Transformer for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2606.16474/) (2026)
- [Why does Deep Learning Improve Visual SLAM?](https://mlsanity.com/paper/2607.06023/) (2026)
- [Monocular Visual Odometry without Calibration or Test-time Optimization](https://mlsanity.com/paper/2510.03348/) (2025)
- [Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos](https://mlsanity.com/paper/2512.10956/) (2025)
- [H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors](https://mlsanity.com/paper/2605.14963/) (2026)
- [Con-DSO: Learning Short-Horizon Consistency Priors for RGB-D Direct Sparse Odometry](https://mlsanity.com/paper/2605.27952/) (2026)
- [DL-VINS-Factory: A Modular Framework for Learned Visual Front-Ends in Visual-Inertial SLAM](https://mlsanity.com/paper/2607.01757/) (2026)
- [DROID-ANCHOR: Odometry-Anchored Recurrent Metric Depth Estimation](https://mlsanity.com/paper/2607.17058/) (2026)
- [Spotter: Efficient Urban Visual Localization via Geo-Referenced Facade Landmarks in GPS-Degraded Environments](https://mlsanity.com/paper/2608.23290/) (2026)
- [KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry](https://mlsanity.com/paper/2608.24544/) (2026)
- [MomentBA: Second-order Spatial Moments for Anisotropic Correspondence Uncertainty in Differentiable Bundle Adjustment](https://mlsanity.com/paper/2609.13691/) (2026)

## Abstract (arXiv)

We propose the MAC-VO, a novel learning-based stereo VO that leverages the learned metrics-aware matching uncertainty for dual purposes: selecting keypoint and weighing the residual in pose graph optimization. Compared to traditional geometric methods prioritizing texture-affluent features like edges, our keypoint selector employs the learned uncertainty to filter out the low-quality features based on global inconsistency. In contrast to the learning-based algorithms that model the scale-agnostic diagonal weight matrix for covariance, we design a metrics-aware covariance model to capture the spatial error during keypoint registration and the correlations between different axes. Integrating this covariance model into pose graph optimization enhances the robustness and reliability of pose estimation, particularly in challenging environments with varying illumination, feature density, and motion patterns. On public benchmark datasets, MAC-VO outperforms existing VO algorithms and even some SLAM algorithms in challenging environments. The covariance map also provides valuable information about the reliability of the estimated poses, which can benefit decision-making for autonomous systems.