# StereoSpike: Depth Learning with a Spiking Neural Network

arXiv: https://arxiv.org/abs/2109.13751 · HTML: https://mlsanity.com/paper/2109.13751/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [Event-Based Angular Velocity Regression with Spiking Networks](https://mlsanity.com/paper/2003.02790/) (2020)
- [SpikeMS: Deep Spiking Neural Network for Motion Segmentation](https://mlsanity.com/paper/2105.06562/) (2021)
- [Realtime Time Synchronized Event-based Stereo](https://mlsanity.com/paper/1803.09025/) (2018)
- [Low-activity supervised convolutional spiking neural networks applied to speech commands recognition](https://mlsanity.com/paper/2011.06846/) (2020)
- [Fast Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/1708.02550/) (2017)
- [Instantaneous Stereo Depth Estimation of Real-World Stimuli with a Neuromorphic Stereo-Vision Setup](https://mlsanity.com/paper/2104.02541/) (2021)

## Built on by (most cited first, 21 of 21)

- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Event-based Video Reconstruction via Potential-assisted Spiking Neural Network](https://mlsanity.com/paper/2201.10943/) (2022)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Event-based Stereo Depth Estimation: A Survey](https://mlsanity.com/paper/2409.17680/) (2024)
- [Spiking Neural Networks for Frame-based and Event-based Single Object Localization](https://mlsanity.com/paper/2206.06506/) (2022)
- [Spiking neural networks for nonlinear regression](https://mlsanity.com/paper/2210.03515/) (2022)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network](https://mlsanity.com/paper/2304.11857/) (2023)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)
- [Single Channel Speech Enhancement Using U-Net Spiking Neural Networks](https://mlsanity.com/paper/2307.14464/) (2023)
- [EventVGGT: Exploring Cross-Modal Distillation for Consistent Event-based Depth Estimation](https://mlsanity.com/paper/2603.09385/) (2026)
- [DERD-Net: Learning Depth from Event-based Ray Densities](https://mlsanity.com/paper/2504.15863/) (2025)
- [MSS-DepthNet: Depth Prediction with Multi-Step Spiking Neural Network](https://mlsanity.com/paper/2211.12156/) (2022)
- [Sparse-firing regularization methods for spiking neural networks with time-to-first spike coding](https://mlsanity.com/paper/2307.13007/) (2023)
- [Event-Driven Neuromorphic Vision Enables Energy-Efficient Visual Place Recognition](https://mlsanity.com/paper/2604.03277/) (2026)
- [Uncertainty Guided Depth Fusion for Spike Camera](https://mlsanity.com/paper/2208.12653/) (2022)
- [Event-based Shape from Polarization with Spiking Neural Networks](https://mlsanity.com/paper/2312.16071/) (2023)
- [SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams](https://mlsanity.com/paper/2505.19487/) (2025)
- [DELTA: Dense Depth from Events and LiDAR using Transformer's Attention](https://mlsanity.com/paper/2505.02593/) (2025)

## Abstract (arXiv)

Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here we solved it using an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a slightly modified U-Net-like encoder-decoder architecture, that we named StereoSpike. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction -- the depth of each pixel -- from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking network. Finally, we show that low firing rates (<10%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.