# Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

arXiv: https://arxiv.org/abs/2003.06696 · HTML: https://mlsanity.com/paper/2003.06696/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2020.

## Built on by (most cited first, 30 of 98)

- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Secrets of Event-Based Optical Flow](https://mlsanity.com/paper/2207.10022/) (2022)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Advancing Spiking Neural Networks towards Deep Residual Learning](https://mlsanity.com/paper/2112.08954/) (2021)
- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation](https://mlsanity.com/paper/2109.04871/) (2021)
- [Dense Continuous-Time Optical Flow from Events and Frames](https://mlsanity.com/paper/2203.13674/) (2022)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation](https://mlsanity.com/paper/2303.07716/) (2023)
- [TMA: Temporal Motion Aggregation for Event-based Optical Flow](https://mlsanity.com/paper/2303.11629/) (2023)
- [Globally-Optimal Contrast Maximisation for Event Cameras](https://mlsanity.com/paper/2206.05127/) (2022)
- [Lightweight Event-based Optical Flow Estimation via Iterative Deblurring](https://mlsanity.com/paper/2211.13726/) (2022)
- [SpikeMS: Deep Spiking Neural Network for Motion Segmentation](https://mlsanity.com/paper/2105.06562/) (2021)
- [Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras](https://mlsanity.com/paper/2112.10591/) (2021)
- [Learning Optical Flow from Event Camera with Rendered Dataset](https://mlsanity.com/paper/2303.11011/) (2023)
- [RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2309.15082/) (2023)
- [Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures](https://mlsanity.com/paper/2103.10592/) (2021)
- [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities](https://mlsanity.com/paper/2211.10754/) (2022)
- [Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation](https://mlsanity.com/paper/2407.10802/) (2024)
- [Fully Spiking Variational Autoencoder](https://mlsanity.com/paper/2110.00375/) (2021)
- [Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision](https://mlsanity.com/paper/2112.03423/) (2021)
- [StereoSpike: Depth Learning with a Spiking Neural Network](https://mlsanity.com/paper/2109.13751/) (2021)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics](https://mlsanity.com/paper/2209.11741/) (2022)
- [EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset](https://mlsanity.com/paper/2312.08220/) (2023)
- [Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer](https://mlsanity.com/paper/2402.04798/) (2024)
- [Event-based Temporally Dense Optical Flow Estimation with Sequential Learning](https://mlsanity.com/paper/2210.01244/) (2022)
- [Towards Anytime Optical Flow Estimation with Event Cameras](https://mlsanity.com/paper/2307.05033/) (2023)

## Abstract (arXiv)

Event-based cameras display great potential for a variety of tasks such as high-speed motion detection and navigation in low-light environments where conventional frame-based cameras suffer critically. This is attributed to their high temporal resolution, high dynamic range, and low-power consumption. However, conventional computer vision methods as well as deep Analog Neural Networks (ANNs) are not suited to work well with the asynchronous and discrete nature of event camera outputs. Spiking Neural Networks (SNNs) serve as ideal paradigms to handle event camera outputs, but deep SNNs suffer in terms of performance due to the spike vanishing phenomenon. To overcome these issues, we present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs without sacrificing the performance. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset. Spike-FlowNet outperforms its corresponding ANN-based method in terms of the optical flow prediction capability while providing significant computational efficiency.