# DH-PTAM: A Deep Hybrid Stereo Events-Frames Parallel Tracking And Mapping System

arXiv: https://arxiv.org/abs/2306.01891 · HTML: https://mlsanity.com/paper/2306.01891/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios](https://mlsanity.com/paper/1709.06310/) (2017)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [Event-based Stereo Visual Odometry](https://mlsanity.com/paper/2007.15548/) (2020)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Semi-Dense 3D Reconstruction with a Stereo Event Camera](https://mlsanity.com/paper/1807.07429/) (2018)
- [Visual-Inertial Mapping with Non-Linear Factor Recovery](https://mlsanity.com/paper/1904.06504/) (2019)
- [A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03638/) (2019)
- [VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM](https://mlsanity.com/paper/2207.01404/) (2022)
- [TUM-VIE: The TUM Stereo Visual-Inertial Event Dataset](https://mlsanity.com/paper/2108.07329/) (2021)
- [ESVIO: Event-based Stereo Visual Inertial Odometry](https://mlsanity.com/paper/2212.13184/) (2022)
- [Multi-Event-Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion](https://mlsanity.com/paper/2207.10494/) (2022)
- [PL-EVIO: Robust Monocular Event-based Visual Inertial Odometry with Point and Line Features](https://mlsanity.com/paper/2209.12160/) (2022)
- [Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events](https://mlsanity.com/paper/2102.13406/) (2021)
- [A Comparison of Modern General-Purpose Visual SLAM Approaches](https://mlsanity.com/paper/2107.07589/) (2021)
- [IBISCape: A Simulated Benchmark for multi-modal SLAM Systems Evaluation in Large-scale Dynamic Environments](https://mlsanity.com/paper/2206.13455/) (2022)
- [Robust GNSS Denied Localization for UAV Using Particle Filter and Visual Odometry](https://mlsanity.com/paper/1910.12121/) (2019)

## Built on by (most cited first, 4 of 4)

- [Deep Event Visual Odometry](https://mlsanity.com/paper/2312.09800/) (2023)
- [Event-based Stereo Depth Estimation: A Survey](https://mlsanity.com/paper/2409.17680/) (2024)
- [EvtSlowTV -- A Large and Diverse Dataset for Event-Based Depth Estimation](https://mlsanity.com/paper/2511.02953/) (2025)
- [SuperEIO: Self-Supervised Event Feature Learning for Event Inertial Odometry](https://mlsanity.com/paper/2503.22963/) (2025)

## Abstract (arXiv)

This paper presents a robust approach for a visual parallel tracking and mapping (PTAM) system that excels in challenging environments. Our proposed method combines the strengths of heterogeneous multi-modal visual sensors, including stereo event-based and frame-based sensors, in a unified reference frame through a novel spatio-temporal synchronization of stereo visual frames and stereo event streams. We employ deep learning-based feature extraction and description for estimation to enhance robustness further. We also introduce an end-to-end parallel tracking and mapping optimization layer complemented by a simple loop-closure algorithm for efficient SLAM behavior. Through comprehensive experiments on both small-scale and large-scale real-world sequences of VECtor and TUM-VIE benchmarks, our proposed method (DH-PTAM) demonstrates superior performance in terms of robustness and accuracy in adverse conditions, especially in large-scale HDR scenarios. Our implementation's research-based Python API is publicly available on GitHub for further research and development: https://github.com/AbanobSoliman/DH-PTAM.