# Wide-Baseline Relative Camera Pose Estimation with Directional Learning

arXiv: https://arxiv.org/abs/2106.03336 · HTML: https://mlsanity.com/paper/2106.03336/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset](https://mlsanity.com/paper/1809.00716/) (2018)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model](https://mlsanity.com/paper/1709.03966/) (2017)
- [Explaining the Ambiguity of Object Detection and 6D Pose From Visual Data](https://mlsanity.com/paper/1812.00287/) (2018)

## Built on by (most cited first, 30 of 49)

- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment](https://mlsanity.com/paper/2306.15667/) (2023)
- [Cameras as Rays: Pose Estimation via Ray Diffusion](https://mlsanity.com/paper/2402.14817/) (2024)
- [Map-free Visual Relocalization: Metric Pose Relative to a Single Image](https://mlsanity.com/paper/2210.05494/) (2022)
- [RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild](https://mlsanity.com/paper/2208.05963/) (2022)
- [RelPose++: Recovering 6D Poses from Sparse-view Observations](https://mlsanity.com/paper/2305.04926/) (2023)
- [LEAP: Liberate Sparse-view 3D Modeling from Camera Poses](https://mlsanity.com/paper/2310.01410/) (2023)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [Map-Relative Pose Regression for Visual Re-Localization](https://mlsanity.com/paper/2404.09884/) (2024)
- [Matrix3D: Large Photogrammetry Model All-in-One](https://mlsanity.com/paper/2502.07685/) (2025)
- [The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTs](https://mlsanity.com/paper/2208.08988/) (2022)
- [Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs](https://mlsanity.com/paper/2312.07246/) (2023)
- [PlaneFormers: From Sparse View Planes to 3D Reconstruction](https://mlsanity.com/paper/2208.04307/) (2022)
- [Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences](https://mlsanity.com/paper/2404.06337/) (2024)
- [Virtual Correspondence: Humans as a Cue for Extreme-View Geometry](https://mlsanity.com/paper/2206.08365/) (2022)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon](https://mlsanity.com/paper/2309.16634/) (2023)
- [FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation](https://mlsanity.com/paper/2403.03221/) (2024)
- [GPR-Net: Multi-view Layout Estimation via a Geometry-aware Panorama Registration Network](https://mlsanity.com/paper/2210.11419/) (2022)
- [GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding](https://mlsanity.com/paper/2201.08131/) (2022)
- [Graph-CoVis: GNN-based Multi-view Panorama Global Pose Estimation](https://mlsanity.com/paper/2304.13201/) (2023)
- [What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction](https://mlsanity.com/paper/2112.04481/) (2021)
- [Can Generative Video Models Help Pose Estimation?](https://mlsanity.com/paper/2412.16155/) (2024)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [Extreme Rotation Estimation in the Wild](https://mlsanity.com/paper/2411.07096/) (2024)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Emergent Extreme-View Geometry in 3D Foundation Models](https://mlsanity.com/paper/2511.22686/) (2025)
- [Visual Correspondence Hallucination](https://mlsanity.com/paper/2106.09711/) (2021)
- [RelMobNet: End-to-end relative camera pose estimation using a robust two-stage training](https://mlsanity.com/paper/2202.12838/) (2022)
- [3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction](https://mlsanity.com/paper/2411.00543/) (2024)

## Abstract (arXiv)

Modern deep learning techniques that regress the relative camera pose between two images have difficulty dealing with challenging scenarios, such as large camera motions resulting in occlusions and significant changes in perspective that leave little overlap between images. These models continue to struggle even with the benefit of large supervised training datasets. To address the limitations of these models, we take inspiration from techniques that show regressing keypoint locations in 2D and 3D can be improved by estimating a discrete distribution over keypoint locations. Analogously, in this paper we explore improving camera pose regression by instead predicting a discrete distribution over camera poses. To realize this idea, we introduce DirectionNet, which estimates discrete distributions over the 5D relative pose space using a novel parameterization to make the estimation problem tractable. Specifically, DirectionNet factorizes relative camera pose, specified by a 3D rotation and a translation direction, into a set of 3D direction vectors. Since 3D directions can be identified with points on the sphere, DirectionNet estimates discrete distributions on the sphere as its output. We evaluate our model on challenging synthetic and real pose estimation datasets constructed from Matterport3D and InteriorNet. Promising results show a near 50% reduction in error over direct regression methods.