# Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures

arXiv: https://arxiv.org/abs/2103.10592 · HTML: https://mlsanity.com/paper/2103.10592/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation](https://mlsanity.com/paper/1804.01306/) (2018)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Single Image Optical Flow Estimation with an Event Camera](https://mlsanity.com/paper/2004.00347/) (2020)
- [Coherent Motion Segmentation in Moving Camera Videos using Optical Flow Orientations](https://mlsanity.com/paper/1511.01619/) (2015)

## Built on by (most cited first, 25 of 25)

- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [Learning Optical Flow from Event Camera with Rendered Dataset](https://mlsanity.com/paper/2303.11011/) (2023)
- [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities](https://mlsanity.com/paper/2211.10754/) (2022)
- [Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics](https://mlsanity.com/paper/2209.11741/) (2022)
- [Event-based Temporally Dense Optical Flow Estimation with Sequential Learning](https://mlsanity.com/paper/2210.01244/) (2022)
- [FE-Fusion-VPR: Attention-based Multi-Scale Network Architecture for Visual Place Recognition by Fusing Frames and Events](https://mlsanity.com/paper/2211.12244/) (2022)
- [EV-MGRFlowNet: Motion-Guided Recurrent Network for Unsupervised Event-based Optical Flow with Hybrid Motion-Compensation Loss](https://mlsanity.com/paper/2305.07853/) (2023)
- [EMatch: A Unified Framework for Event-based Optical Flow and Stereo Matching](https://mlsanity.com/paper/2407.21735/) (2024)
- [Instance-Level Moving Object Segmentation from a Single Image with Events](https://mlsanity.com/paper/2502.12975/) (2025)
- [Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach](https://mlsanity.com/paper/2507.03765/) (2025)
- [Intelligent Sensing-to-Action for Robust Autonomy at the Edge: Opportunities and Challenges](https://mlsanity.com/paper/2502.02692/) (2025)
- [DeblurSR: Event-Based Motion Deblurring Under the Spiking Representation](https://mlsanity.com/paper/2303.08977/) (2023)
- [Best of Both Worlds: Hybrid SNN-ANN Architecture for Event-based Optical Flow Estimation](https://mlsanity.com/paper/2306.02960/) (2023)
- [Learning Efficient Meshflow and Optical Flow from Event Cameras](https://mlsanity.com/paper/2510.04111/) (2025)
- [Ev-Edge: Efficient Execution of Event-based Vision Algorithms on Commodity Edge Platforms](https://mlsanity.com/paper/2403.15717/) (2024)
- [ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition](https://mlsanity.com/paper/2409.01564/) (2024)
- [Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application](https://mlsanity.com/paper/2207.10720/) (2022)
- [SHIRE: Enhancing Sample Efficiency using Human Intuition in REinforcement Learning](https://mlsanity.com/paper/2409.09990/) (2024)
- [FEDORA: Flying Event Dataset fOr Reactive behAvior](https://mlsanity.com/paper/2305.14392/) (2023)
- [Spike-TBR: a Noise Resilient Neuromorphic Event Representation](https://mlsanity.com/paper/2506.04817/) (2025)
- [Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow](https://mlsanity.com/paper/2505.05089/) (2025)
- [EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras](https://mlsanity.com/paper/2506.06596/) (2025)

## Abstract (arXiv)

Standard frame-based cameras that sample light intensity frames are heavily impacted by motion blur for high-speed motion and fail to perceive scene accurately when the dynamic range is high. Event-based cameras, on the other hand, overcome these limitations by asynchronously detecting the variation in individual pixel intensities. However, event cameras only provide information about pixels in motion, leading to sparse data. Hence, estimating the overall dense behavior of pixels is difficult. To address such issues associated with the sensors, we present Fusion-FlowNet, a sensor fusion framework for energy-efficient optical flow estimation using both frame- and event-based sensors, leveraging their complementary characteristics. Our proposed network architecture is also a fusion of Spiking Neural Networks (SNNs) and Analog Neural Networks (ANNs) where each network is designed to simultaneously process asynchronous event streams and regular frame-based images, respectively. Our network is end-to-end trained using unsupervised learning to avoid expensive video annotations. The method generalizes well across distinct environments (rapid motion and challenging lighting conditions) and demonstrates state-of-the-art optical flow prediction on the Multi-Vehicle Stereo Event Camera (MVSEC) dataset. Furthermore, our network offers substantial savings in terms of the number of network parameters and computational energy cost.