# Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation

arXiv: https://arxiv.org/abs/2110.07742 · HTML: https://mlsanity.com/paper/2110.07742/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 79 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [EV-SegNet: Semantic Segmentation for Event-based Cameras](https://mlsanity.com/paper/1811.12039/) (2018)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch](https://mlsanity.com/paper/2010.01729/) (2020)
- [Supervised learning based on temporal coding in spiking neural networks](https://mlsanity.com/paper/1606.08165/) (2016)
- [DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks](https://mlsanity.com/paper/2008.03658/) (2020)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding](https://mlsanity.com/paper/2003.11741/) (2020)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)

## Built on by (most cited first, 30 of 55)

- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANN](https://mlsanity.com/paper/2305.19868/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities](https://mlsanity.com/paper/2211.10754/) (2022)
- [Scaling Up Dynamic Graph Representation Learning via Spiking Neural Networks](https://mlsanity.com/paper/2208.10364/) (2022)
- [Backpropagation with Biologically Plausible Spatio-Temporal Adjustment For Training Deep Spiking Neural Networks](https://mlsanity.com/paper/2110.08858/) (2021)
- [Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation](https://mlsanity.com/paper/2412.14587/) (2024)
- [Spiking Neural Networks for Frame-based and Event-based Single Object Localization](https://mlsanity.com/paper/2206.06506/) (2022)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Brain-inspired Multilayer Perceptron with Spiking Neurons](https://mlsanity.com/paper/2203.14679/) (2022)
- [Spike Calibration: Fast and Accurate Conversion of Spiking Neural Network for Object Detection and Segmentation](https://mlsanity.com/paper/2207.02702/) (2022)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras](https://mlsanity.com/paper/2304.10211/) (2023)
- [Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network](https://mlsanity.com/paper/2304.11857/) (2023)
- [LoAS: Fully Temporal-Parallel Dataflow for Dual-Sparse Spiking Neural Networks](https://mlsanity.com/paper/2407.14073/) (2024)
- [Brain-Inspired Online Adaptation for Remote Sensing with Spiking Neural Network](https://mlsanity.com/paper/2409.02146/) (2024)
- [EventRPG: Event Data Augmentation with Relevance Propagation Guidance](https://mlsanity.com/paper/2403.09274/) (2024)
- [EvSegSNN: Neuromorphic Semantic Segmentation for Event Data](https://mlsanity.com/paper/2406.14178/) (2024)
- [A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks](https://mlsanity.com/paper/2305.19306/) (2023)
- [Finding Meaning in Points: Weakly Supervised Semantic Segmentation for Event Cameras](https://mlsanity.com/paper/2407.11216/) (2024)
- [Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach](https://mlsanity.com/paper/2507.03765/) (2025)
- [SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations](https://mlsanity.com/paper/2211.09928/) (2022)
- [Hybrid Spiking Neural Network Fine-tuning for Hippocampus Segmentation](https://mlsanity.com/paper/2302.07328/) (2023)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)
- [Automotive Object Detection via Learning Sparse Events by Spiking Neurons](https://mlsanity.com/paper/2307.12900/) (2023)
- [MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.10210/) (2024)
- [CEIA: CLIP-Based Event-Image Alignment for Open-World Event-Based Understanding](https://mlsanity.com/paper/2407.06611/) (2024)
- [Hoyer regularizer is all you need for ultra low-latency spiking neural networks](https://mlsanity.com/paper/2212.10170/) (2022)
- [Biologically-inspired Semi-supervised Semantic Segmentation for Biomedical Imaging](https://mlsanity.com/paper/2412.03192/) (2024)

## Abstract (arXiv)

Spiking Neural Networks (SNNs) have recently emerged as the low-power alternative to Artificial Neural Networks (ANNs) because of their sparse, asynchronous, and binary event-driven processing. Due to their energy efficiency, SNNs have a high possibility of being deployed for real-world, resource-constrained systems such as autonomous vehicles and drones. However, owing to their non-differentiable and complex neuronal dynamics, most previous SNN optimization methods have been limited to image recognition. In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons. Specifically, we first investigate two representative SNN optimization techniques for recognition tasks (i.e., ANN-SNN conversion and surrogate gradient learning) on semantic segmentation datasets. We observe that, when converted from ANNs, SNNs suffer from high latency and low performance due to the spatial variance of features. Therefore, we directly train networks with surrogate gradient learning, resulting in lower latency and higher performance than ANN-SNN conversion. Moreover, we redesign two fundamental ANN segmentation architectures (i.e., Fully Convolutional Networks and DeepLab) for the SNN domain. We conduct experiments on two public semantic segmentation benchmarks including the PASCAL VOC2012 dataset and the DDD17 event-based dataset. In addition to showing the feasibility of SNNs for semantic segmentation, we show that SNNs can be more robust and energy-efficient compared to their ANN counterparts in this domain.