# DiffPoseNet: Direct Differentiable Camera Pose Estimation

arXiv: https://arxiv.org/abs/2203.11174 · HTML: https://mlsanity.com/paper/2203.11174/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning](https://mlsanity.com/paper/1709.06841/) (2017)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [Attacking Optical Flow](https://mlsanity.com/paper/1910.10053/) (2019)
- [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization](https://mlsanity.com/paper/2007.14628/) (2020)
- [EVPropNet: Detecting Drones By Finding Propellers For Mid-Air Landing And Following](https://mlsanity.com/paper/2106.15045/) (2021)
- [Measuring robustness of Visual SLAM](https://mlsanity.com/paper/1910.04755/) (2019)
- [Vertical Landing for Micro Air Vehicles using Event-Based Optical Flow](https://mlsanity.com/paper/1702.00061/) (2017)
- [Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients](https://mlsanity.com/paper/1805.06641/) (2018)
- [Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization](https://mlsanity.com/paper/2002.11826/) (2020)

## Built on by (most cited first, 27 of 27)

- [Detector-Free Structure from Motion](https://mlsanity.com/paper/2306.15669/) (2023)
- [Learning How To Robustly Estimate Camera Pose in Endoscopic Videos](https://mlsanity.com/paper/2304.08023/) (2023)
- [Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction](https://mlsanity.com/paper/2604.08542/) (2026)
- [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry](https://mlsanity.com/paper/2409.09479/) (2024)
- [iSLAM: Imperative SLAM](https://mlsanity.com/paper/2306.07894/) (2023)
- [Motion and Structure from Event-based Normal Flow](https://mlsanity.com/paper/2407.12239/) (2024)
- [NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields](https://mlsanity.com/paper/2312.13471/) (2023)
- [Graph-CoVis: GNN-based Multi-view Panorama Global Pose Estimation](https://mlsanity.com/paper/2304.13201/) (2023)
- [Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments](https://mlsanity.com/paper/2304.07250/) (2023)
- [MC-NeRF: Multi-Camera Neural Radiance Fields for Multi-Camera Image Acquisition Systems](https://mlsanity.com/paper/2309.07846/) (2023)
- [Learning Normal Flow Directly From Event Neighborhoods](https://mlsanity.com/paper/2412.11284/) (2024)
- [DINO-VO: A Feature-based Visual Odometry Leveraging a Visual Foundation Model](https://mlsanity.com/paper/2507.13145/) (2025)
- [CUAHN-VIO: Content-and-Uncertainty-Aware Homography Network for Visual-Inertial Odometry](https://mlsanity.com/paper/2208.13935/) (2022)
- [Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes](https://mlsanity.com/paper/2309.08588/) (2023)
- [Geometry-Aware Feature Matching for Large-Scale Structure from Motion](https://mlsanity.com/paper/2409.02310/) (2024)
- [AlignDiff: Learning Physically-Grounded Camera Alignment via Diffusion](https://mlsanity.com/paper/2503.21581/) (2025)
- [Exploiting Motion Prior for Accurate Pose Estimation of Dashboard Cameras](https://mlsanity.com/paper/2409.18673/) (2024)
- [Robust Two-View Geometry Estimation with Implicit Differentiation](https://mlsanity.com/paper/2410.17983/) (2024)
- [Globally Optimal Solution to the Generalized Relative Pose Estimation Problem using Affine Correspondences](https://mlsanity.com/paper/2512.17188/) (2025)
- [A Minimal Solver for Relative Pose Estimation with Unknown Focal Length from Two Affine Correspondences](https://mlsanity.com/paper/2512.22833/) (2025)
- [MVOFormer: Flow-Semantic Transformer for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2606.16474/) (2026)
- [DIME-Net: Neural Network-Based Dynamic Intrinsic Parameter Rectification for Cameras with Optical Image Stabilization System](https://mlsanity.com/paper/2303.11307/) (2023)
- [T-Graph: Enhancing Sparse-view Camera Pose Estimation by Pairwise Translation Graph](https://mlsanity.com/paper/2505.01207/) (2025)
- [Odometry Without Correspondence from Inertially Constrained Ruled Surfaces](https://mlsanity.com/paper/2512.00327/) (2025)
- [Real-time Motion Segmentation with Event-based Normal Flow](https://mlsanity.com/paper/2602.20790/) (2026)

## Abstract (arXiv)

Current deep neural network approaches for camera pose estimation rely on scene structure for 3D motion estimation, but this decreases the robustness and thereby makes cross-dataset generalization difficult. In contrast, classical approaches to structure from motion estimate 3D motion utilizing optical flow and then compute depth. Their accuracy, however, depends strongly on the quality of the optical flow. To avoid this issue, direct methods have been proposed, which separate 3D motion from depth estimation but compute 3D motion using only image gradients in the form of normal flow. In this paper, we introduce a network NFlowNet, for normal flow estimation which is used to enforce robust and direct constraints. In particular, normal flow is used to estimate relative camera pose based on the cheirality (depth positivity) constraint. We achieve this by formulating the optimization problem as a differentiable cheirality layer, which allows for end-to-end learning of camera pose. We perform extensive qualitative and quantitative evaluation of the proposed DiffPoseNet's sensitivity to noise and its generalization across datasets. We compare our approach to existing state-of-the-art methods on KITTI, TartanAir, and TUM-RGBD datasets.