# Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Loss

arXiv: https://arxiv.org/abs/2103.00760 · HTML: https://mlsanity.com/paper/2103.00760/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [SfM-Net: Learning of Structure and Motion from Video](https://mlsanity.com/paper/1704.07804/) (2017)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth](https://mlsanity.com/paper/1904.07087/) (2019)
- [Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation](https://mlsanity.com/paper/1907.10303/) (2019)
- [Sparse Depth Enhanced Direct Thermal-infrared SLAM Beyond the Visible Spectrum](https://mlsanity.com/paper/1902.10892/) (2019)
- [DeepTIO: A Deep Thermal-Inertial Odometry with Visual Hallucination](https://mlsanity.com/paper/1909.07231/) (2019)
- [Camera Exposure Control for Robust Robot Vision with Noise-Aware Image Quality Assessment](https://mlsanity.com/paper/1907.12646/) (2019)
- [Multi-Spectral Visual Odometry without Explicit Stereo Matching](https://mlsanity.com/paper/1908.08814/) (2019)

## Built on by (most cited first, 16 of 16)

- [Complementary Random Masking for RGB-Thermal Semantic Segmentation](https://mlsanity.com/paper/2303.17386/) (2023)
- [Maximizing Self-supervision from Thermal Image for Effective Self-supervised Learning of Depth and Ego-motion](https://mlsanity.com/paper/2201.04387/) (2022)
- [RIDERS: Radar-Infrared Depth Estimation for Robust Sensing](https://mlsanity.com/paper/2402.02067/) (2024)
- [MonoTher-Depth: Enhancing Thermal Depth Estimation via Confidence-Aware Distillation](https://mlsanity.com/paper/2504.16127/) (2025)
- [Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios](https://mlsanity.com/paper/2402.11826/) (2024)
- [ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Images](https://mlsanity.com/paper/2403.09871/) (2024)
- [Deep Depth Estimation from Thermal Image: Dataset, Benchmark, and Challenges](https://mlsanity.com/paper/2503.22060/) (2025)
- [Thermal Chameleon: Task-Adaptive Tone-mapping for Radiometric Thermal-Infrared images](https://mlsanity.com/paper/2410.18340/) (2024)
- [Bridging Spectral-wise and Multi-spectral Depth Estimation via Geometry-guided Contrastive Learning](https://mlsanity.com/paper/2503.00793/) (2025)
- [SLAM in the Dark: Self-Supervised Learning of Pose, Depth and Loop-Closure from Thermal Images](https://mlsanity.com/paper/2502.18932/) (2025)
- [RGB-Thermal Infrared Fusion for Robust Depth Estimation in Complex Environments](https://mlsanity.com/paper/2503.04821/) (2025)
- [MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis](https://mlsanity.com/paper/2511.22997/) (2025)
- [SA-DNet: A on-demand semantic object registration network adapting to non-rigid deformation](https://mlsanity.com/paper/2210.09900/) (2022)
- [Thermal Image Refinement with Depth Estimation using Recurrent Networks for Monocular ORB-SLAM3](https://mlsanity.com/paper/2603.14998/) (2026)
- [DarkVGGT: Seeing Through Darkness Using Thermal Geometry](https://mlsanity.com/paper/2606.11326/) (2026)
- [TIDY: Thermal Infrared Image Denoising via Wavelet Domain Entropy and Directional Stripe Index](https://mlsanity.com/paper/2606.19813/) (2026)

## Abstract (arXiv)

A thermal camera can robustly capture thermal radiation images under harsh light conditions such as night scenes, tunnels, and disaster scenarios. However, despite this advantage, neither depth nor ego-motion estimation research for the thermal camera have not been actively explored so far. In this paper, we propose a self-supervised learning method for depth and ego-motion estimation from thermal images. The proposed method exploits multi-spectral consistency that consists of temperature and photometric consistency loss. The temperature consistency loss provides a fundamental self-supervisory signal by reconstructing clipped and colorized thermal images. Additionally, we design a differentiable forward warping module that can transform the coordinate system of the estimated depth map and relative pose from thermal camera to visible camera. Based on the proposed module, the photometric consistency loss can provide complementary self-supervision to networks. Networks trained with the proposed method robustly estimate the depth and pose from monocular thermal video under low-light and even zero-light conditions. To the best of our knowledge, this is the first work to simultaneously estimate both depth and ego-motion from monocular thermal video in a self-supervised manner.