# HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events

arXiv: https://arxiv.org/abs/2411.18658 · HTML: https://mlsanity.com/paper/2411.18658/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2024.

## Built on by (most cited first, 4 of 4)

- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection](https://mlsanity.com/paper/2505.12908/) (2025)
- [CETUS: Causal Event-Driven Temporal Modeling With Unified Variable-Rate Scheduling](https://mlsanity.com/paper/2509.13784/) (2025)
- [FATE: Pillar Encoding and Frequency-Aware Training for Event-Based Object Detection](https://mlsanity.com/paper/2606.17334/) (2026)

## Abstract (arXiv)

Combining the complementary benefits of frames and events has been widely used for object detection in challenging scenarios. However, most object detection methods use two independent Artificial Neural Network (ANN) branches, limiting cross-modality information interaction across the two visual streams and encountering challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using frames and events. Technically, we first present a novel semantic-enhanced self-attention mechanism that strengthens the correlation between image encoding tokens within the ANN Transformer branch for better performance. Then, we design a Spiking Swin Transformer branch to model temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks for cross-modality information interaction. The results demonstrate that our HDI-Former outperforms eleven state-of-the-art methods and our four baselines by a large margin. Our SNN branch also shows comparable performance to the ANN with the same architecture while consuming 10.57× less energy on the DSEC-Detection dataset. Our open-source code is available in the supplementary material.