# Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach

arXiv: https://arxiv.org/abs/2507.03765 · HTML: https://mlsanity.com/paper/2507.03765/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 4 Jul 2025.

## Builds on (most cited first, 28 of 28)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [EV-SegNet: Semantic Segmentation for Event-based Cameras](https://mlsanity.com/paper/1811.12039/) (2018)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/2110.07742/) (2021)
- [EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation](https://mlsanity.com/paper/2111.12341/) (2021)
- [Hierarchical Neural Memory Network for Low Latency Event Processing](https://mlsanity.com/paper/2305.17852/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation](https://mlsanity.com/paper/2109.01801/) (2021)
- [CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation](https://mlsanity.com/paper/2307.15942/) (2023)
- [A Spiking Neural Network for Image Segmentation](https://mlsanity.com/paper/2106.08921/) (2021)
- [Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures](https://mlsanity.com/paper/2103.10592/) (2021)
- [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities](https://mlsanity.com/paper/2211.10754/) (2022)
- [Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision](https://mlsanity.com/paper/2112.03423/) (2021)
- [Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera](https://mlsanity.com/paper/2309.09297/) (2023)
- [Are High-Resolution Event Cameras Really Needed?](https://mlsanity.com/paper/2203.14672/) (2022)
- [Event-based Synthetic Aperture Imaging with a Hybrid Network](https://mlsanity.com/paper/2103.02376/) (2021)
- [Event-based Simultaneous Localization and Mapping: A Comprehensive Survey](https://mlsanity.com/paper/2304.09793/) (2023)
- [Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction](https://mlsanity.com/paper/2403.07346/) (2024)
- [Hybrid Spiking Neural Network Fine-tuning for Hippocampus Segmentation](https://mlsanity.com/paper/2302.07328/) (2023)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)

## Built on by (most cited first, 6 of 6)

- [Reliable Brain Tumor Segmentation Based on Spiking Neural Networks with Efficient Training](https://mlsanity.com/paper/2601.16652/) (2026)
- [DIMOS: Disentangling Instance-level Moving Object Segmentation](https://mlsanity.com/paper/2606.12826/) (2026)
- [Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing](https://mlsanity.com/paper/2607.09143/) (2026)
- [LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction](https://mlsanity.com/paper/2609.25803/) (2026)
- [The Shape of Events: Edge-Based Inductive Biases via Cross-Domain Distillation](https://mlsanity.com/paper/2609.30478/) (2026)
- [Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation](https://mlsanity.com/paper/2610.17800/) (2026)

## Abstract (arXiv)

Event cameras have recently been introduced into image semantic segmentation, owing to their high temporal resolution and other advantageous properties. However, existing event-based semantic segmentation methods often fail to fully exploit the complementary information provided by frames and events, resulting in complex training strategies and increased computational costs. To address these challenges, we propose an efficient hybrid framework for image semantic segmentation, comprising a Spiking Neural Network branch for events and an Artificial Neural Network branch for frames. Specifically, we introduce three specialized modules to facilitate the interaction between these two branches: the Adaptive Temporal Weighting (ATW) Injector, the Event-Driven Sparse (EDS) Injector, and the Channel Selection Fusion (CSF) module. The ATW Injector dynamically integrates temporal features from event data into frame features, enhancing segmentation accuracy by leveraging critical dynamic temporal information. The EDS Injector effectively combines sparse event data with rich frame features, ensuring precise temporal and spatial information alignment. The CSF module selectively merges these features to optimize segmentation performance. Experimental results demonstrate that our framework not only achieves state-of-the-art accuracy across the DDD17-Seg, DSEC-Semantic, and M3ED-Semantic datasets but also significantly reduces energy consumption, achieving a 65% reduction on the DSEC-Semantic dataset.