# Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth

arXiv: https://arxiv.org/abs/2203.01557 · HTML: https://mlsanity.com/paper/2203.01557/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning](https://mlsanity.com/paper/1709.06841/) (2017)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Attention-Based Multimodal Fusion for Video Description](https://mlsanity.com/paper/1701.03126/) (2017)
- [Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth](https://mlsanity.com/paper/1904.07087/) (2019)
- [Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling](https://mlsanity.com/paper/2007.10983/) (2020)
- [Self-Supervised Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2005.06136/) (2020)
- [Selective Sensor Fusion for Neural Visual-Inertial Odometry](https://mlsanity.com/paper/1903.01534/) (2019)
- [Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry](https://mlsanity.com/paper/1904.01892/) (2019)
- [DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints](https://mlsanity.com/paper/1906.11435/) (2019)
- [GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks](https://mlsanity.com/paper/1809.05786/) (2018)
- [Deep Surface Normal Estimation with Hierarchical RGB-D Fusion](https://mlsanity.com/paper/1904.03405/) (2019)
- [Pose Graph Optimization for Unsupervised Monocular Visual Odometry](https://mlsanity.com/paper/1903.06315/) (2019)
- [Self-supervised Visual-LiDAR Odometry with Flip Consistency](https://mlsanity.com/paper/2101.01322/) (2021)
- [Two Stream Networks for Self-Supervised Ego-Motion Estimation](https://mlsanity.com/paper/1910.01764/) (2019)
- [Unsupervised Learning of Camera Pose with Compositional Re-estimation](https://mlsanity.com/paper/2001.06479/) (2020)

## Built on by (most cited first, 6 of 6)

- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation](https://mlsanity.com/paper/2304.07728/) (2023)
- [EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting Ego-Motion Rigidity](https://mlsanity.com/paper/2309.01296/) (2023)
- [Exchanging-based Multimodal Fusion with Transformer](https://mlsanity.com/paper/2309.02190/) (2023)
- [UNO: Unified Self-Supervised Monocular Odometry for Platform-Agnostic Deployment](https://mlsanity.com/paper/2506.07013/) (2025)
- [Discriminately Treating Motion Components Evolves Joint Depth and Ego-Motion Learning](https://mlsanity.com/paper/2511.01502/) (2025)

## Abstract (arXiv)

In existing self-supervised depth and ego-motion estimation methods, ego-motion estimation is usually limited to only leveraging RGB information. Recently, several methods have been proposed to further improve the accuracy of self-supervised ego-motion estimation by fusing information from other modalities, e.g., depth, acceleration, and angular velocity. However, they rarely focus on how different fusion strategies affect performance. In this paper, we investigate the effect of different fusion strategies for ego-motion estimation and propose a new framework for self-supervised learning of depth and ego-motion estimation, which performs ego-motion estimation by leveraging RGB and inferred depth information in a Multi-Layer Fusion manner. As a result, we have achieved state-of-the-art performance among learning-based methods on the KITTI odometry benchmark. Detailed studies on the design choices of leveraging inferred depth information and fusion strategies have also been carried out, which clearly demonstrate the advantages of our proposed framework.