# Frequency-Adaptive Low-Latency Object Detection Using Events and Frames

arXiv: https://arxiv.org/abs/2412.04149 · HTML: https://mlsanity.com/paper/2412.04149/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [State Space Models for Event Cameras](https://mlsanity.com/paper/2402.15584/) (2024)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras](https://mlsanity.com/paper/1805.07931/) (2018)
- [Frame-Event Alignment and Fusion Network for High Frame Rate Tracking](https://mlsanity.com/paper/2305.15688/) (2023)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [SODFormer: Streaming Object Detection with Transformer Using Events and Frames](https://mlsanity.com/paper/2308.04047/) (2023)
- [Scene Adaptive Sparse Transformer for Event-based Object Detection](https://mlsanity.com/paper/2404.01882/) (2024)
- [A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/2104.10719/) (2021)
- [Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection](https://mlsanity.com/paper/2407.12582/) (2024)
- [Motion Robust High-Speed Light-Weighted Object Detection With Event Camera](https://mlsanity.com/paper/2208.11602/) (2022)

## Built on by (most cited first, 7 of 7)

- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [Sensor Generalization for Adaptive Sensing in Event-based Object Detection via Joint Distribution Training](https://mlsanity.com/paper/2602.23357/) (2026)
- [Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE](https://mlsanity.com/paper/2604.11140/) (2026)
- [M^2C-EvDet: Multi-Domain Multi-Order Cross-Modal Knowledge Distillation for Event-based Object Detection](https://mlsanity.com/paper/2606.24248/) (2026)
- [UMSS: Towards Unsupervised Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2607.12372/) (2026)
- [SkyEV: RGB-Event UAV detection and tracking dataset and baseline](https://mlsanity.com/paper/2607.18747/) (2026)
- [Neuromorphic Object Detection: An In-Depth Study and Future Directions](https://mlsanity.com/paper/2607.23576/) (2026)

## Abstract (arXiv)

Fusing Events and RGB images for object detection leverages the robustness of Event cameras in adverse environments and the rich semantic information provided by RGB cameras. However, two critical mismatches: low-latency Events textitvs.~high-latency RGB frames; temporally sparse labels in training textitvs.~continuous flow in inference, significantly hinder the high-frequency fusion-based object detection. To address these challenges, we propose the Frequency-Adaptive Low-Latency Object Detector (FAOD). FAOD aligns low-frequency RGB frames with high-frequency Events through an Align Module, which reinforces cross-modal style and spatial proximity to address the Event-RGB Mismatch. We further propose a training strategy, Time Shift, which enforces the module to align the prediction from temporally shifted Event-RGB pairs and their original representation, that is, consistent with Event-aligned annotations. This strategy enables the network to use high-frequency Event data as the primary reference while treating low-frequency RGB images as supplementary information, retaining the low-latency nature of the Event stream toward high-frequency detection. Furthermore, we observe that these corrected Event-RGB pairs demonstrate better generalization from low training frequency to higher inference frequencies compared to using Event data alone. Extensive experiments on the PKU-DAVIS-SOD and DSEC-Detection datasets demonstrate that our FAOD achieves SOTA performance. Specifically, in the PKU-DAVIS-SOD Dataset, FAOD achieves 9.8 points improvement in terms of the mAP in fully paired Event-RGB data with only a quarter of the parameters compared to SODFormer, and even maintains robust performance (only a 3 points drop in mAP) under 80× Event-RGB frequency mismatch.