# Deep Auxiliary Learning for Visual Localization and Odometry

arXiv: https://arxiv.org/abs/1803.03642 · HTML: https://mlsanity.com/paper/1803.03642/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2018.

## Built on by (most cited first, 30 of 59)

- [GNeRF: GAN-based Neural Radiance Field without Posed Camera](https://mlsanity.com/paper/2103.15606/) (2021)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [DFNet: Enhance Absolute Pose Regression with Direct Feature Matching](https://mlsanity.com/paper/2204.00559/) (2022)
- [Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection](https://mlsanity.com/paper/2112.04628/) (2021)
- [Direct-PoseNet: Absolute Pose Regression with Photometric Consistency](https://mlsanity.com/paper/2104.04073/) (2021)
- [MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction](https://mlsanity.com/paper/2110.01401/) (2021)
- [Neural Refinement for Absolute Pose Regression with Feature Synthesis](https://mlsanity.com/paper/2303.10087/) (2023)
- [CoordiNet: uncertainty-aware pose regressor for reliable vehicle localization](https://mlsanity.com/paper/2103.10796/) (2021)
- [MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks](https://mlsanity.com/paper/2203.06801/) (2022)
- [Learning Camera Localization via Dense Scene Matching](https://mlsanity.com/paper/2103.16792/) (2021)
- [Camera Pose Auto-Encoders for Improving Pose Regression](https://mlsanity.com/paper/2207.05530/) (2022)
- [DytanVO: Joint Refinement of Visual Odometry and Motion Segmentation in Dynamic Environments](https://mlsanity.com/paper/2209.08430/) (2022)
- [HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization with Transformer](https://mlsanity.com/paper/2305.03595/) (2023)
- [Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba](https://mlsanity.com/paper/2405.06116/) (2024)
- [Continual Learning for Image-Based Camera Localization](https://mlsanity.com/paper/2108.09112/) (2021)
- [A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization](https://mlsanity.com/paper/2403.19412/) (2024)
- [JigsawGAN: Auxiliary Learning for Solving Jigsaw Puzzles with Generative Adversarial Networks](https://mlsanity.com/paper/2101.07555/) (2021)
- [HR-APR: APR-agnostic Framework with Uncertainty Estimation and Hierarchical Refinement for Camera Relocalisation](https://mlsanity.com/paper/2402.14371/) (2024)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Coarse-to-Fine Multi-Scene Pose Regression with Transformers](https://mlsanity.com/paper/2308.11783/) (2023)
- [BronchoTrack: Airway Lumen Tracking for Branch-Level Bronchoscopic Localization](https://mlsanity.com/paper/2402.12763/) (2024)
- [Learning Neural Volumetric Pose Features for Camera Localization](https://mlsanity.com/paper/2403.12800/) (2024)
- [Paying Attention to Activation Maps in Camera Pose Regression](https://mlsanity.com/paper/2103.11477/) (2021)
- [A Critical Analysis of Image-based Camera Pose Estimation Techniques](https://mlsanity.com/paper/2201.05816/) (2022)
- [DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy](https://mlsanity.com/paper/2403.01683/) (2024)
- [H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awareness](https://mlsanity.com/paper/2204.13852/) (2022)
- [Modality-invariant Visual Odometry for Embodied Vision](https://mlsanity.com/paper/2305.00348/) (2023)
- [Sample-Level Weighting for Multi-Task Learning with Auxiliary Tasks](https://mlsanity.com/paper/2306.04519/) (2023)
- [Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments](https://mlsanity.com/paper/2304.07250/) (2023)
- [RaLF: Flow-based Global and Metric Radar Localization in LiDAR Maps](https://mlsanity.com/paper/2309.09875/) (2023)

## Abstract (arXiv)

Localization is an indispensable component of a robot's autonomy stack that enables it to determine where it is in the environment, essentially making it a precursor for any action execution or planning. Although convolutional neural networks have shown promising results for visual localization, they are still grossly outperformed by state-of-the-art local feature-based techniques. In this work, we propose VLocNet, a new convolutional neural network architecture for 6-DoF global pose regression and odometry estimation from consecutive monocular images. Our multitask model incorporates hard parameter sharing, thus being compact and enabling real-time inference, in addition to being end-to-end trainable. We propose a novel loss function that utilizes auxiliary learning to leverage relative pose information during training, thereby constraining the search space to obtain consistent pose estimates. We evaluate our proposed VLocNet on indoor as well as outdoor datasets and show that even our single task model exceeds the performance of state-of-the-art deep architectures for global localization, while achieving competitive performance for visual odometry estimation. Furthermore, we present extensive experimental evaluations utilizing our proposed Geometric Consistency Loss that show the effectiveness of multitask learning and demonstrate that our model is the first deep learning technique to be on par with, and in some cases outperforms state-of-the-art SIFT-based approaches.