# Distilling Knowledge From a Deep Pose Regressor Network

arXiv: https://arxiv.org/abs/1908.00858 · HTML: https://mlsanity.com/paper/1908.00858/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2019.

## Built on by (most cited first, 30 of 34)

- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectors](https://mlsanity.com/paper/2206.09500/) (2022)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Deep Learning for Inertial Positioning: A Survey](https://mlsanity.com/paper/2303.03757/) (2023)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [Continual Learning for Image-Based Camera Localization](https://mlsanity.com/paper/2108.09112/) (2021)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Deep Visual Odometry with Events and Frames](https://mlsanity.com/paper/2309.09947/) (2023)
- [Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks](https://mlsanity.com/paper/2104.07196/) (2021)
- [Tiny Object Tracking: A Large-scale Dataset and A Baseline](https://mlsanity.com/paper/2202.05659/) (2022)
- [Accelerating Molecular Graph Neural Networks via Knowledge Distillation](https://mlsanity.com/paper/2306.14818/) (2023)
- [Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression](https://mlsanity.com/paper/2104.03164/) (2021)
- [Rethinking Generic Camera Models for Deep Single Image Camera Calibration to Recover Rotation and Fisheye Distortion](https://mlsanity.com/paper/2111.12927/) (2021)
- [Unsupervised Domain Expansion for Visual Categorization](https://mlsanity.com/paper/2104.00233/) (2021)
- [Synthetic data generation method for data-free knowledge distillation in regression neural networks](https://mlsanity.com/paper/2301.04338/) (2023)
- [Continual Learning in Predictive Autoscaling](https://mlsanity.com/paper/2307.15941/) (2023)
- [Performance Evaluation of Semi-supervised Learning Frameworks for Multi-Class Weed Detection](https://mlsanity.com/paper/2403.03390/) (2024)
- [SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators](https://mlsanity.com/paper/2603.20410/) (2026)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction](https://mlsanity.com/paper/2208.07232/) (2022)
- [SFT-KD-Recon: Learning a Student-friendly Teacher for Knowledge Distillation in Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/2304.05057/) (2023)
- [OCHID-Fi: Occlusion-Robust Hand Pose Estimation in 3D via RF-Vision](https://mlsanity.com/paper/2308.10146/) (2023)
- [TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation](https://mlsanity.com/paper/2402.14340/) (2024)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Knowledge distillation with error-correcting transfer learning for wind power prediction](https://mlsanity.com/paper/2204.00649/) (2022)
- [Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2211.15951/) (2022)

## Abstract (arXiv)

This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on "dark knowledge" for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.