# Cross-Modal Semi-Dense 6-DoF Tracking of an Event Camera in Challenging Conditions

arXiv: https://arxiv.org/abs/2401.08043 · HTML: https://mlsanity.com/paper/2401.08043/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 69 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 28 of 28)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields](https://mlsanity.com/paper/2210.13641/) (2022)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios](https://mlsanity.com/paper/1709.06310/) (2017)
- [Event-based Stereo Visual Odometry](https://mlsanity.com/paper/2007.15548/) (2020)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Focus Is All You Need: Loss Functions For Event-based Vision](https://mlsanity.com/paper/1904.07235/) (2019)
- [Continuous-Time Visual-Inertial Odometry for Event Cameras](https://mlsanity.com/paper/1702.07389/) (2017)
- [Event-based, 6-DOF Camera Tracking from Photometric Depth Maps](https://mlsanity.com/paper/1607.03468/) (2016)
- [VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM](https://mlsanity.com/paper/2207.01404/) (2022)
- [DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions](https://mlsanity.com/paper/2202.02556/) (2022)
- [Exploring Event Camera-based Odometry for Planetary Robots](https://mlsanity.com/paper/2204.05880/) (2022)
- [Globally Optimal Contrast Maximisation for Event-based Motion Estimation](https://mlsanity.com/paper/2002.10686/) (2020)
- [Globally-Optimal Contrast Maximisation for Event Cameras](https://mlsanity.com/paper/2206.05127/) (2022)
- [IDOL: A Framework for IMU-DVS Odometry using Lines](https://mlsanity.com/paper/2008.05749/) (2020)
- [VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals](https://mlsanity.com/paper/2104.06789/) (2021)
- [Real-Time Panoramic Tracking for Event Cameras](https://mlsanity.com/paper/1703.05161/) (2017)
- [Visual Odometry with an Event Camera Using Continuous Ray Warping and Volumetric Contrast Maximization](https://mlsanity.com/paper/2107.03011/) (2021)
- [Canny-VO: Visual Odometry with RGB-D Cameras based on Geometric 3D-2D Edge Alignment](https://mlsanity.com/paper/2012.08228/) (2020)
- [Neuromorphic Visual Odometry System for Intelligent Vehicle Application with Bio-inspired Vision Sensor](https://mlsanity.com/paper/1909.02490/) (2019)

## Built on by (most cited first, 9 of 9)

- [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025)
- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [GS-EVT: Cross-Modal Event Camera Tracking based on Gaussian Splatting](https://mlsanity.com/paper/2409.19228/) (2024)
- [EVIT: Event-based Visual-Inertial Tracking in Semi-Dense Maps Using Windowed Nonlinear Optimization](https://mlsanity.com/paper/2408.01370/) (2024)
- [Event-Based Visual Odometry on Non-Holonomic Ground Vehicles](https://mlsanity.com/paper/2401.09331/) (2024)
- [A Survey on Event-driven 3D Reconstruction: Development under Different Categories](https://mlsanity.com/paper/2503.19753/) (2025)
- [AsynEIO: Asynchronous Monocular Event-Inertial Odometry Using Gaussian Process Regression](https://mlsanity.com/paper/2411.12175/) (2024)
- [A Label-Free and Non-Monotonic Metric for Evaluating Denoising in Event Cameras](https://mlsanity.com/paper/2406.08909/) (2024)
- [LEAR: Learning Edge-Aware Representations for Event-to-LiDAR Localization](https://mlsanity.com/paper/2603.01839/) (2026)

## Abstract (arXiv)

Vision-based localization is a cost-effective and thus attractive solution for many intelligent mobile platforms. However, its accuracy and especially robustness still suffer from low illumination conditions, illumination changes, and aggressive motion. Event-based cameras are bio-inspired visual sensors that perform well in HDR conditions and have high temporal resolution, and thus provide an interesting alternative in such challenging scenarios. While purely event-based solutions currently do not yet produce satisfying mapping results, the present work demonstrates the feasibility of purely event-based tracking if an alternative sensor is permitted for mapping. The method relies on geometric 3D-2D registration of semi-dense maps and events, and achieves highly reliable and accurate cross-modal tracking results. Practically relevant scenarios are given by depth camera-supported tracking or map-based localization with a semi-dense map prior created by a regular image-based visual SLAM or structure-from-motion system. Conventional edge-based 3D-2D alignment is extended by a novel polarity-aware registration that makes use of signed time-surface maps (STSM) obtained from event streams. We furthermore introduce a novel culling strategy for occluded points. Both modifications increase the speed of the tracker and its robustness against occlusions or large view-point variations. The approach is validated on many real datasets covering the above-mentioned challenging conditions, and compared against similar solutions realised with regular cameras.