# Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow

arXiv: https://arxiv.org/abs/2503.06992 · HTML: https://mlsanity.com/paper/2503.06992/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better](https://mlsanity.com/paper/1908.03826/) (2019)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Attention-Based Multimodal Fusion for Video Description](https://mlsanity.com/paper/1701.03126/) (2017)
- [DDFlow: Learning Optical Flow with Unlabeled Data Distillation](https://mlsanity.com/paper/1902.09145/) (2019)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)

## Built on by (most cited first, 5 of 5)

- [4D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimation](https://mlsanity.com/paper/2511.18416/) (2025)
- [Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes](https://mlsanity.com/paper/2510.10577/) (2025)
- [TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation](https://mlsanity.com/paper/2603.23487/) (2026)
- [Engram-E2VID: Reference-Based Event-to-Video Reconstruction via Generative Activation of Appearance Engrams](https://mlsanity.com/paper/2608.05728/) (2026)
- [Static in Frames, Dynamic in Events: Rethinking Features in Event Cameras as Motion Cues](https://mlsanity.com/paper/2608.11075/) (2026)

## Abstract (arXiv)

High-dynamic scene optical flow is a challenging task, which suffers spatial blur and temporal discontinuous motion due to large displacement in frame imaging, thus deteriorating the spatiotemporal feature of optical flow. Typically, existing methods mainly introduce event camera to directly fuse the spatiotemporal features between the two modalities. However, this direct fusion is ineffective, since there exists a large gap due to the heterogeneous data representation between frame and event modalities. To address this issue, we explore a common-latent space as an intermediate bridge to mitigate the modality gap. In this work, we propose a novel common spatiotemporal fusion between frame and event modalities for high-dynamic scene optical flow, including visual boundary localization and motion correlation fusion. Specifically, in visual boundary localization, we figure out that frame and event share the similar spatiotemporal gradients, whose similarity distribution is consistent with the extracted boundary distribution. This motivates us to design the common spatiotemporal gradient to constrain the reference boundary localization. In motion correlation fusion, we discover that the frame-based motion possesses spatially dense but temporally discontinuous correlation, while the event-based motion has spatially sparse but temporally continuous correlation. This inspires us to use the reference boundary to guide the complementary motion knowledge fusion between the two modalities. Moreover, common spatiotemporal fusion can not only relieve the cross-modal feature discrepancy, but also make the fusion process interpretable for dense and continuous optical flow. Extensive experiments have been performed to verify the superiority of the proposed method.