# Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application

arXiv: https://arxiv.org/abs/2207.10720 · HTML: https://mlsanity.com/paper/2207.10720/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization](https://mlsanity.com/paper/1903.07414/) (2019)
- [Fast Optical Flow using Dense Inverse Search](https://mlsanity.com/paper/1603.03590/) (2016)
- [Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception](https://mlsanity.com/paper/1807.10936/) (2018)
- [Distance Surface for Event-Based Optical Flow](https://mlsanity.com/paper/2003.12680/) (2020)
- [Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures](https://mlsanity.com/paper/2103.10592/) (2021)
- [Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation](https://mlsanity.com/paper/1508.05151/) (2015)
- [Spiking Optical Flow for Event-based Sensors Using IBM's TrueNorth Neurosynaptic System](https://mlsanity.com/paper/1710.09820/) (2017)
- [Real-Time Optical flow-based Video Stabilization for Unmanned Aerial Vehicles](https://mlsanity.com/paper/1701.03572/) (2017)

## Built on by (most cited first, 2 of 2)

- [Event-based vision on FPGAs -- a survey](https://mlsanity.com/paper/2407.08356/) (2024)
- [Low Latency Visual Inertial Odometry with On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs](https://mlsanity.com/paper/2406.13345/) (2024)

## Abstract (arXiv)

Optical flow computation with frame-based cameras provides high accuracy but the speed is limited either by the model size of the algorithm or by the frame rate of the camera. This makes it inadequate for high-speed applications. Event cameras provide continuous asynchronous event streams overcoming the frame-rate limitation. However, the algorithms for processing the data either borrow frame like setup limiting the speed or suffer from lower accuracy. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow while maintaining a low error rate. Our bio-mimetic network is validated with the MVSEC dataset showing 19% error degradation at 4x speed up. We then demonstrate the system with a high-speed drone flight scenario where a high-speed event camera computes the flow even before the optical camera sees the drone making it suited for applications like tracking and segmentation. This work shows the fundamental trade-offs in frame-based processing may be overcome by fusing data from other modalities.