# Optical flow estimation from event-based cameras and spiking neural networks

arXiv: https://arxiv.org/abs/2302.06492 · HTML: https://mlsanity.com/paper/2302.06492/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 35)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity](https://mlsanity.com/paper/2207.03620/) (2022)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Efficient Neuromorphic Signal Processing with Loihi 2](https://mlsanity.com/paper/2111.03746/) (2021)
- [Secrets of Event-Based Optical Flow](https://mlsanity.com/paper/2207.10022/) (2022)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [Object Detection with Spiking Neural Networks on Automotive Event Data](https://mlsanity.com/paper/2205.04339/) (2022)
- [Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/2110.07742/) (2021)
- [CED: Color Event Camera Dataset](https://mlsanity.com/paper/1904.10772/) (2019)
- [EVIMO2: An Event Camera Dataset for Motion Segmentation, Optical Flow, Structure from Motion, and Visual Inertial Odometry in Indoor Scenes with Monocular or Stereo Algorithms](https://mlsanity.com/paper/2205.03467/) (2022)
- [Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures](https://mlsanity.com/paper/2103.10592/) (2021)
- [Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks](https://mlsanity.com/paper/2104.12579/) (2021)

## Built on by (most cited first, 30 of 33)

- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [State Space Models for Event Cameras](https://mlsanity.com/paper/2402.15584/) (2024)
- [Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flow](https://mlsanity.com/paper/2303.05214/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation](https://mlsanity.com/paper/2303.07716/) (2023)
- [Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation](https://mlsanity.com/paper/2407.10802/) (2024)
- [Event Camera Data Dense Pre-training](https://mlsanity.com/paper/2311.11533/) (2023)
- [Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer](https://mlsanity.com/paper/2402.04798/) (2024)
- [EventVL: Understand Event Streams via Multimodal Large Language Model](https://mlsanity.com/paper/2501.13707/) (2025)
- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [Event-based Optical Flow on Neuromorphic Processor: ANN vs. SNN Comparison based on Activation Sparsification](https://mlsanity.com/paper/2407.20421/) (2024)
- [Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera](https://mlsanity.com/paper/2307.06003/) (2023)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)
- [Learning Normal Flow Directly From Event Neighborhoods](https://mlsanity.com/paper/2412.11284/) (2024)
- [Vector-Symbolic Architecture for Event-Based Optical Flow](https://mlsanity.com/paper/2405.08300/) (2024)
- [Learning Efficient Meshflow and Optical Flow from Event Cameras](https://mlsanity.com/paper/2510.04111/) (2025)
- [Event-Driven Neuromorphic Vision Enables Energy-Efficient Visual Place Recognition](https://mlsanity.com/paper/2604.03277/) (2026)
- [E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization](https://mlsanity.com/paper/2510.12753/) (2025)
- [EventDrive: Event Cameras for Vision-Language Driving Intelligence](https://mlsanity.com/paper/2606.18242/) (2026)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [Enhanced Neuromorphic Semantic Segmentation Latency through Stream Event](https://mlsanity.com/paper/2502.18982/) (2025)
- [SCFlow2: Plug-and-Play Object Pose Refiner with Shape-Constraint Scene Flow](https://mlsanity.com/paper/2504.09160/) (2025)
- [Spike-TBR: a Noise Resilient Neuromorphic Event Representation](https://mlsanity.com/paper/2506.04817/) (2025)
- [Neuromorphic Optical Tracking and Imaging of Randomly Moving Targets through Strongly Scattering Media](https://mlsanity.com/paper/2501.03874/) (2025)
- [ST-FlowNet: An Efficient Spiking Neural Network for Event-Based Optical Flow Estimation](https://mlsanity.com/paper/2503.10195/) (2025)
- [Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow](https://mlsanity.com/paper/2506.07878/) (2025)
- [NeuroLiDAR: Adaptive Frame Rate Depth Sensing via Neuromorphic Event-LiDAR Fusion](https://mlsanity.com/paper/2605.16805/) (2026)
- [LC-Flow: Learning Local Continuous Optical Flow and Confidence from events](https://mlsanity.com/paper/2605.24604/) (2026)

## Abstract (arXiv)

Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or "spikes", when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates.