# RGB-Event Fusion for Moving Object Detection in Autonomous Driving

arXiv: https://arxiv.org/abs/2209.08323 · HTML: https://mlsanity.com/paper/2209.08323/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [Specificity-preserving RGB-D Saliency Detection](https://mlsanity.com/paper/2108.08162/) (2021)
- [Semi-Dense 3D Reconstruction with a Stereo Event Camera](https://mlsanity.com/paper/1807.07429/) (2018)
- [Actions as Moving Points](https://mlsanity.com/paper/2001.04608/) (2020)
- [Finding Action Tubes](https://mlsanity.com/paper/1411.6031/) (2014)
- [R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package](https://mlsanity.com/paper/2109.07982/) (2021)

## Built on by (most cited first, 30 of 40)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection](https://mlsanity.com/paper/2407.12582/) (2024)
- [Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera](https://mlsanity.com/paper/2309.09297/) (2023)
- [Event-Based Eye Tracking. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.11770/) (2024)
- [Enhancing Traffic Object Detection in Variable Illumination with RGB-Event Fusion](https://mlsanity.com/paper/2311.00436/) (2023)
- [SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events](https://mlsanity.com/paper/2309.12842/) (2023)
- [TUMTraf Event: Calibration and Fusion Resulting in a Dataset for Roadside Event-Based and RGB Cameras](https://mlsanity.com/paper/2401.08474/) (2024)
- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation](https://mlsanity.com/paper/2407.10703/) (2024)
- [Event-Free Moving Object Segmentation from Moving Ego Vehicle](https://mlsanity.com/paper/2305.00126/) (2023)
- [Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation](https://mlsanity.com/paper/2510.00681/) (2025)
- [Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes](https://mlsanity.com/paper/2403.04562/) (2024)
- [Frequency-Adaptive Low-Latency Object Detection Using Events and Frames](https://mlsanity.com/paper/2412.04149/) (2024)
- [Tracking Any Point with Frame-Event Fusion Network at High Frame Rate](https://mlsanity.com/paper/2409.11953/) (2024)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios](https://mlsanity.com/paper/2508.10704/) (2025)
- [EF-Calib: Spatiotemporal Calibration of Event- and Frame-Based Cameras Using Continuous-Time Trajectories](https://mlsanity.com/paper/2405.17278/) (2024)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Deep Event-based Object Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2405.03995/) (2024)
- [Hardware, Algorithms, and Applications of the Neuromorphic Vision Sensor: a Review](https://mlsanity.com/paper/2504.08588/) (2025)
- [Steering Prediction via a Multi-Sensor System for Autonomous Racing](https://mlsanity.com/paper/2409.19356/) (2024)
- [FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion](https://mlsanity.com/paper/2503.19739/) (2025)
- [Event-based Civil Infrastructure Visual Defect Detection: ev-CIVIL Dataset and Benchmark](https://mlsanity.com/paper/2504.05679/) (2025)
- [EventDrive: Event Cameras for Vision-Language Driving Intelligence](https://mlsanity.com/paper/2606.18242/) (2026)
- [EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion](https://mlsanity.com/paper/2312.16933/) (2023)
- [RGB-Event Fusion with Self-Attention for Collision Prediction](https://mlsanity.com/paper/2505.04258/) (2025)
- [WD-DETR: Wavelet Denoising-Enhanced Real-Time Object Detection Transformer for Robot Perception with Event Cameras](https://mlsanity.com/paper/2506.09098/) (2025)

## Abstract (arXiv)

Moving Object Detection (MOD) is a critical vision task for successfully achieving safe autonomous driving. Despite plausible results of deep learning methods, most existing approaches are only frame-based and may fail to reach reasonable performance when dealing with dynamic traffic participants. Recent advances in sensor technologies, especially the Event camera, can naturally complement the conventional camera approach to better model moving objects. However, event-based works often adopt a pre-defined time window for event representation, and simply integrate it to estimate image intensities from events, neglecting much of the rich temporal information from the available asynchronous events. Therefore, from a new perspective, we propose RENet, a novel RGB-Event fusion Network, that jointly exploits the two complementary modalities to achieve more robust MOD under challenging scenarios for autonomous driving. Specifically, we first design a temporal multi-scale aggregation module to fully leverage event frames from both the RGB exposure time and larger intervals. Then we introduce a bi-directional fusion module to attentively calibrate and fuse multi-modal features. To evaluate the performance of our network, we carefully select and annotate a sub-MOD dataset from the commonly used DSEC dataset. Extensive experiments demonstrate that our proposed method performs significantly better than the state-of-the-art RGB-Event fusion alternatives. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/RENet.