# Deep Visual Odometry with Events and Frames

arXiv: https://arxiv.org/abs/2309.09947 · HTML: https://mlsanity.com/paper/2309.09947/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 30 of 36)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios](https://mlsanity.com/paper/1709.06310/) (2017)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Event-based Stereo Visual Odometry](https://mlsanity.com/paper/2007.15548/) (2020)
- [Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1811.08230/) (2018)
- [EV-SegNet: Semantic Segmentation for Event-based Cameras](https://mlsanity.com/paper/1811.12039/) (2018)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [Semi-Dense 3D Reconstruction with a Stereo Event Camera](https://mlsanity.com/paper/1807.07429/) (2018)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction](https://mlsanity.com/paper/2005.08605/) (2020)
- [A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03638/) (2019)
- [DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions](https://mlsanity.com/paper/2202.02556/) (2022)
- [Distilling Knowledge From a Deep Pose Regressor Network](https://mlsanity.com/paper/1908.00858/) (2019)

## Built on by (most cited first, 13 of 13)

- [EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Images](https://mlsanity.com/paper/2405.20224/) (2024)
- [Full-DoF Egomotion Estimation for Event Cameras Using Geometric Solvers](https://mlsanity.com/paper/2503.03307/) (2025)
- [BlinkTrack: Feature Tracking over 80 FPS via Events and Images](https://mlsanity.com/paper/2409.17981/) (2024)
- [A Linear N-Point Solver for Structure and Motion from Asynchronous Tracks](https://mlsanity.com/paper/2507.22733/) (2025)
- [Learning to Remove Lens Flare in Event Camera](https://mlsanity.com/paper/2512.09016/) (2025)
- [E2EGS: Event-to-Edge Gaussian Splatting for Pose-Free 3D Reconstruction](https://mlsanity.com/paper/2603.14684/) (2026)
- [RGB-Event Fusion with Self-Attention for Collision Prediction](https://mlsanity.com/paper/2505.04258/) (2025)
- [Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videos](https://mlsanity.com/paper/2505.18899/) (2025)
- [E-TIDE: Fast, Structure-Preserving Motion Forecasting from Event Sequences](https://mlsanity.com/paper/2603.27757/) (2026)
- [Robust Monocular Visual Odometry using Curriculum Learning](https://mlsanity.com/paper/2411.13438/) (2024)
- [SuperEIO: Self-Supervised Event Feature Learning for Event Inertial Odometry](https://mlsanity.com/paper/2503.22963/) (2025)
- [Minimal Solvers for Full-DoF Motion Estimation from Asynchronous Differential SfM](https://mlsanity.com/paper/2606.09218/) (2026)
- [DAPEVO: Deep Adaptive Patch Frame-Event Visual Odometry](https://mlsanity.com/paper/2609.30947/) (2026)

## Abstract (arXiv)

Visual Odometry (VO) is crucial for autonomous robotic navigation, especially in GPS-denied environments like planetary terrains. To improve robustness, recent model-based VO systems have begun combining standard and event-based cameras. While event cameras excel in low-light and high-speed motion, standard cameras provide dense and easier-to-track features. However, the field of image- and event-based VO still predominantly relies on model-based methods and is yet to fully integrate recent image-only advancements leveraging end-to-end learning-based architectures. Seamlessly integrating the two modalities remains challenging due to their different nature, one asynchronous, the other not, limiting the potential for a more effective image- and event-based VO. We introduce RAMP-VO, the first end-to-end learned image- and event-based VO system. It leverages novel Recurrent, Asynchronous, and Massively Parallel (RAMP) encoders capable of fusing asynchronous events with image data, providing 8x faster inference and 33% more accurate predictions than existing solutions. Despite being trained only in simulation, RAMP-VO outperforms previous methods on the newly introduced Apollo and Malapert datasets, and on existing benchmarks, where it improves image- and event-based methods by 58.8% and 30.6%, paving the way for robust and asynchronous VO in space.