# Dense Continuous-Time Optical Flow from Events and Frames

arXiv: https://arxiv.org/abs/2203.13674 · HTML: https://mlsanity.com/paper/2203.13674/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Continuous-Time Visual-Inertial Odometry for Event Cameras](https://mlsanity.com/paper/1702.07389/) (2017)
- [High-Resolution Optical Flow from 1D Attention and Correlation](https://mlsanity.com/paper/2104.13918/) (2021)
- [A Fusion Approach for Multi-Frame Optical Flow Estimation](https://mlsanity.com/paper/1810.10066/) (2018)
- [Single Image Optical Flow Estimation with an Event Camera](https://mlsanity.com/paper/2004.00347/) (2020)
- [STaRFlow: A SpatioTemporal Recurrent Cell for Lightweight Multi-Frame Optical Flow Estimation](https://mlsanity.com/paper/2007.05481/) (2020)
- [ProFlow: Learning to Predict Optical Flow](https://mlsanity.com/paper/1806.00800/) (2018)
- [CNN-based Patch Matching for Optical Flow with Thresholded Hinge Embedding Loss](https://mlsanity.com/paper/1607.08064/) (2016)
- [PatchBatch: a Batch Augmented Loss for Optical Flow](https://mlsanity.com/paper/1512.01815/) (2015)

## Built on by (most cited first, 30 of 46)

- [Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation](https://mlsanity.com/paper/2407.10802/) (2024)
- [Towards Anytime Optical Flow Estimation with Event Cameras](https://mlsanity.com/paper/2307.05033/) (2023)
- [Event-based Continuous Color Video Decompression from Single Frames](https://mlsanity.com/paper/2312.00113/) (2023)
- [Data-Driven Feature Tracking for Event Cameras With and Without Frames](https://mlsanity.com/paper/2211.12826/) (2022)
- [Temporal Event Stereo via Joint Learning with Stereoscopic Flow](https://mlsanity.com/paper/2407.10831/) (2024)
- [Table tennis ball spin estimation with an event camera](https://mlsanity.com/paper/2404.09870/) (2024)
- [ETAP: Event-based Tracking of Any Point](https://mlsanity.com/paper/2412.00133/) (2024)
- [TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion](https://mlsanity.com/paper/2505.03116/) (2025)
- [Flow-Anything: Learning Real-World Optical Flow Estimation from Large-Scale Single-view Images](https://mlsanity.com/paper/2506.07740/) (2025)
- [Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras](https://mlsanity.com/paper/2502.19630/) (2025)
- [Future Optical Flow Prediction Improves Robot Control & Video Generation](https://mlsanity.com/paper/2601.10781/) (2026)
- [Tracking Any Point with Frame-Event Fusion Network at High Frame Rate](https://mlsanity.com/paper/2409.11953/) (2024)
- [Secrets of Edge-Informed Contrast Maximization for Event-Based Vision](https://mlsanity.com/paper/2409.14611/) (2024)
- [Instance-Level Moving Object Segmentation from a Single Image with Events](https://mlsanity.com/paper/2502.12975/) (2025)
- [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025)
- [EDCFlow: Exploring Temporally Dense Difference Maps for Event-based Optical Flow Estimation](https://mlsanity.com/paper/2506.03512/) (2025)
- [Learning Normal Flow Directly From Event Neighborhoods](https://mlsanity.com/paper/2412.11284/) (2024)
- [Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow](https://mlsanity.com/paper/2503.06992/) (2025)
- [BlinkTrack: Feature Tracking over 80 FPS via Events and Images](https://mlsanity.com/paper/2409.17981/) (2024)
- [MATE: Motion-Augmented Temporal Consistency for Event-based Point Tracking](https://mlsanity.com/paper/2412.01300/) (2024)
- [Spatially-guided Temporal Aggregation for Robust Event-RGB Optical Flow Estimation](https://mlsanity.com/paper/2501.00838/) (2025)
- [Learning Efficient Meshflow and Optical Flow from Event Cameras](https://mlsanity.com/paper/2510.04111/) (2025)
- [ResFlow: Fine-tuning Residual Optical Flow for Event-based High Temporal Resolution Motion Estimation](https://mlsanity.com/paper/2412.09105/) (2024)
- [Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes](https://mlsanity.com/paper/2510.10577/) (2025)
- [EMoTive: Event-guided Trajectory Modeling for 3D Motion Estimation](https://mlsanity.com/paper/2503.11371/) (2025)
- [Perturbed State Space Feature Encoders for Optical Flow with Event Cameras](https://mlsanity.com/paper/2504.10669/) (2025)
- [E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization](https://mlsanity.com/paper/2510.12753/) (2025)
- [Learning to Remove Lens Flare in Event Camera](https://mlsanity.com/paper/2512.09016/) (2025)
- [Event-based Optical Marker Systems: A survey](https://mlsanity.com/paper/2504.20736/) (2025)
- [DESSERT: Diffusion-based Event-driven Single-frame Synthesis via Residual Training](https://mlsanity.com/paper/2512.17323/) (2025)

## Abstract (arXiv)

We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized B'ezier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use B'ezier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the B'ezier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To the best of our knowledge, our model is the first method that can regress dense pixel trajectories from event data. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments not only suggest that our method successfully predicts pixel trajectories in continuous time but also that it is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public.