# REMOTE: Real-time Ego-motion Tracking for Various Endoscopes via Multimodal Visual Feature Learning

arXiv: https://arxiv.org/abs/2501.18124 · HTML: https://mlsanity.com/paper/2501.18124/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [AtLoc: Attention Guided Camera Localization](https://mlsanity.com/paper/1909.03557/) (2019)
- [VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization](https://mlsanity.com/paper/1702.06521/) (2017)
- [FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation](https://mlsanity.com/paper/2103.04524/) (2021)
- [Bimodal Camera Pose Prediction for Endoscopy](https://mlsanity.com/paper/2204.04968/) (2022)
- [VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization](https://mlsanity.com/paper/2003.07289/) (2020)
- [OffsetNet: Deep Learning for Localization in the Lung using Rendered Images](https://mlsanity.com/paper/1809.05645/) (2018)
- [BronchoPose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation](https://mlsanity.com/paper/2204.11982/) (2022)
- [Adversarial Networks for Camera Pose Regression and Refinement](https://mlsanity.com/paper/1903.06646/) (2019)

## Abstract (arXiv)

Real-time ego-motion tracking for endoscope is a significant task for efficient navigation and robotic automation of endoscopy. In this paper, a novel framework is proposed to perform real-time ego-motion tracking for endoscope. Firstly, a multi-modal visual feature learning network is proposed to perform relative pose prediction, in which the motion feature from the optical flow, the scene features and the joint feature from two adjacent observations are all extracted for prediction. Due to more correlation information in the channel dimension of the concatenated image, a novel feature extractor is designed based on an attention mechanism to integrate multi-dimensional information from the concatenation of two continuous frames. To extract more complete feature representation from the fused features, a novel pose decoder is proposed to predict the pose transformation from the concatenated feature map at the end of the framework. At last, the absolute pose of endoscope is calculated based on relative poses. The experiment is conducted on three datasets of various endoscopic scenes and the results demonstrate that the proposed method outperforms state-of-the-art methods. Besides, the inference speed of the proposed method is over 30 frames per second, which meets the real-time requirement. The project page is here: remote-bmxs.netlify.app