# Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection

arXiv: https://arxiv.org/abs/2407.12582 · HTML: https://mlsanity.com/paper/2407.12582/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 61 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016)
- [Imbalance Problems in Object Detection: A Review](https://mlsanity.com/paper/1909.00169/) (2019)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection](https://mlsanity.com/paper/2004.08515/) (2020)

## Built on by (most cited first, 28 of 28)

- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation](https://mlsanity.com/paper/2510.00681/) (2025)
- [Frequency-Adaptive Low-Latency Object Detection Using Events and Frames](https://mlsanity.com/paper/2412.04149/) (2024)
- [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios](https://mlsanity.com/paper/2508.10704/) (2025)
- [BlinkTrack: Feature Tracking over 80 FPS via Events and Images](https://mlsanity.com/paper/2409.17981/) (2024)
- [Unleashing the Temporal Potential of Stereo Event Cameras for Continuous-Time 3D Object Detection](https://mlsanity.com/paper/2508.02288/) (2025)
- [URNet: Uncertainty-aware Refinement Network for Event-based Stereo Depth Estimation](https://mlsanity.com/paper/2509.18184/) (2025)
- [Event-aided Semantic Scene Completion](https://mlsanity.com/paper/2502.02334/) (2025)
- [Enhancing 3D Semantic Scene Completion with a Refinement Module](https://mlsanity.com/paper/2512.18363/) (2025)
- [Steering Prediction via a Multi-Sensor System for Autonomous Racing](https://mlsanity.com/paper/2409.19356/) (2024)
- [EvAnimate: Event-conditioned Image-to-Video Generation for Human Animation](https://mlsanity.com/paper/2503.18552/) (2025)
- [EventDrive: Event Cameras for Vision-Language Driving Intelligence](https://mlsanity.com/paper/2606.18242/) (2026)
- [Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection](https://mlsanity.com/paper/2505.12908/) (2025)
- [WD-DETR: Wavelet Denoising-Enhanced Real-Time Object Detection Transformer for Robot Perception with Event Cameras](https://mlsanity.com/paper/2506.09098/) (2025)
- [Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detection](https://mlsanity.com/paper/2509.03872/) (2025)
- [TUMTraf EMOT: Event-Based Multi-Object Tracking Dataset and Baseline for Traffic Scenarios](https://mlsanity.com/paper/2512.14595/) (2025)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [RoEL: Robust Event-based 3D Line Reconstruction](https://mlsanity.com/paper/2602.18258/) (2026)
- [Energy-Aware Imitation Learning for Steering Prediction Using Events and Frames](https://mlsanity.com/paper/2603.28008/) (2026)
- [SegRGB-X: General RGB-X Semantic Segmentation Model](https://mlsanity.com/paper/2603.28023/) (2026)
- [CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection](https://mlsanity.com/paper/2606.29136/) (2026)
- [Neuromorphic Object Detection: An In-Depth Study and Future Directions](https://mlsanity.com/paper/2607.23576/) (2026)
- [FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learning](https://mlsanity.com/paper/2608.16523/) (2026)
- [Efficient and Robust Absolute Pose Estimation via Gravity-Prior-Driven Transformation Decoupling and Pose Refinement](https://mlsanity.com/paper/2609.00713/) (2026)
- [IT-TextFusion: Iterative Text-Image Interaction with Text-Guided Residual Refinement for Degradation-Aware Image Fusion](https://mlsanity.com/paper/2609.01092/) (2026)
- [Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation](https://mlsanity.com/paper/2610.17800/) (2026)

## Abstract (arXiv)

In frame-based vision, object detection faces substantial performance degradation under challenging conditions due to the limited sensing capability of conventional cameras. Event cameras output sparse and asynchronous events, providing a potential solution to solve these problems. However, effectively fusing two heterogeneous modalities remains an open issue. In this work, we propose a novel hierarchical feature refinement network for event-frame fusion. The core concept is the design of the coarse-to-fine fusion module, denoted as the cross-modality adaptive feature refinement (CAFR) module. In the initial phase, the bidirectional cross-modality interaction (BCI) part facilitates information bridging from two distinct sources. Subsequently, the features are further refined by aligning the channel-level mean and variance in the two-fold adaptive feature refinement (TAFR) part. We conducted extensive experiments on two benchmarks: the low-resolution PKU-DDD17-Car dataset and the high-resolution DSEC dataset. Experimental results show that our method surpasses the state-of-the-art by an impressive margin of 8.0% on the DSEC dataset. Besides, our method exhibits significantly better robustness (69.5% versus 38.7%) when introducing 15 different corruption types to the frame images. The code can be found at the link (https://github.com/HuCaoFighting/FRN).