# MemFlow: Optical Flow Estimation and Prediction with Memory

arXiv: https://arxiv.org/abs/2404.04808 · HTML: https://mlsanity.com/paper/2404.04808/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 75 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 30 of 55)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation](https://mlsanity.com/paper/2306.01923/) (2023)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [Overcoming a Theoretical Limitation of Self-Attention](https://mlsanity.com/paper/2202.12172/) (2022)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016)

## Built on by (most cited first, 30 of 55)

- [DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework](https://mlsanity.com/paper/2503.14880/) (2025)
- [SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models](https://mlsanity.com/paper/2411.13112/) (2024)
- [VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation](https://mlsanity.com/paper/2502.07531/) (2025)
- [MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion](https://mlsanity.com/paper/2507.08344/) (2025)
- [L4P: Towards Unified Low-Level 4D Vision Perception](https://mlsanity.com/paper/2502.13078/) (2025)
- [TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer](https://mlsanity.com/paper/2506.18904/) (2025)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation](https://mlsanity.com/paper/2412.07160/) (2024)
- [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025)
- [MFTIQ: Multi-Flow Tracker with Independent Matching Quality Estimation](https://mlsanity.com/paper/2411.09551/) (2024)
- [AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction](https://mlsanity.com/paper/2504.01014/) (2025)
- [MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation](https://mlsanity.com/paper/2506.23151/) (2025)
- [Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence](https://mlsanity.com/paper/2607.12477/) (2026)
- [MegaFlow: Zero-Shot Large Displacement Optical Flow](https://mlsanity.com/paper/2603.25739/) (2026)
- [Real-Time Human Action Recognition on Embedded Platforms](https://mlsanity.com/paper/2409.05662/) (2024)
- [GDROS: A Geometry-Guided Dense Registration Framework for Optical-SAR Images under Large Geometric Transformations](https://mlsanity.com/paper/2511.00598/) (2025)
- [GG-SSMs: Graph-Generating State Space Models](https://mlsanity.com/paper/2412.12423/) (2024)
- [Online Dense Point Tracking with Streaming Memory](https://mlsanity.com/paper/2503.06471/) (2025)
- [DynaSplat: Dynamic-Static Gaussian Splatting with Hierarchical Motion Decomposition for Scene Reconstruction](https://mlsanity.com/paper/2506.09836/) (2025)
- [Mem4D: Decoupling Static and Dynamic Memory for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2508.07908/) (2025)
- [Removing Cost Volumes from Optical Flow Estimators](https://mlsanity.com/paper/2510.13317/) (2025)
- [Gaussian Splatting Lucas-Kanade](https://mlsanity.com/paper/2407.11309/) (2024)
- [EMoTive: Event-guided Trajectory Modeling for 3D Motion Estimation](https://mlsanity.com/paper/2503.11371/) (2025)
- [LinkTo-Anime: A 2D Animation Optical Flow Dataset from 3D Model Rendering](https://mlsanity.com/paper/2506.02733/) (2025)
- [MIORe & VAR-MIORe: Benchmarks to Push the Boundaries of Restoration](https://mlsanity.com/paper/2509.06803/) (2025)
- [FLAG-4D: Flow-Guided Local-Global Dual-Deformation Model for 4D Reconstruction](https://mlsanity.com/paper/2602.08558/) (2026)
- [F2HDR: Two-Stage HDR Video Reconstruction via Flow Adapter and Physical Motion Modeling](https://mlsanity.com/paper/2603.14920/) (2026)
- [vSTMD: Visual Motion Detection for Extremely Tiny Target at Various Velocities](https://mlsanity.com/paper/2501.13054/) (2025)
- [BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation](https://mlsanity.com/paper/2503.03256/) (2025)
- [DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding](https://mlsanity.com/paper/2504.00432/) (2025)

## Abstract (arXiv)

Optical flow is a classical task that is important to the vision community. Classical optical flow estimation uses two frames as input, whilst some recent methods consider multiple frames to explicitly model long-range information. The former ones limit their ability to fully leverage temporal coherence along the video sequence; and the latter ones incur heavy computational overhead, typically not possible for real-time flow estimation. Some multi-frame-based approaches even necessitate unseen future frames for current estimation, compromising real-time applicability in safety-critical scenarios. To this end, we present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Furthermore, we integrate resolution-adaptive re-scaling to accommodate diverse video resolutions. Besides, our approach seamlessly extends to the future prediction of optical flow based on past observations. Leveraging effective historical motion aggregation, our method outperforms VideoFlow with fewer parameters and faster inference speed on Sintel and KITTI-15 datasets in terms of generalization performance. At the time of submission, MemFlow also leads in performance on the 1080p Spring dataset. Codes and models will be available at: https://dqiaole.github.io/MemFlow/.