# SplatFlow: Learning Multi-frame Optical Flow via Splatting

arXiv: https://arxiv.org/abs/2306.08887 · HTML: https://mlsanity.com/paper/2306.08887/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 73 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 49)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [Know Your Surroundings: Exploiting Scene Information for Object Tracking](https://mlsanity.com/paper/2003.11014/) (2020)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)

## Built on by (most cited first, 7 of 7)

- [Bolt3D: Generating 3D Scenes in Seconds](https://mlsanity.com/paper/2503.14445/) (2025)
- [SceneTracker: Long-term Scene Flow Estimation Network](https://mlsanity.com/paper/2403.19924/) (2024)
- [StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences](https://mlsanity.com/paper/2311.17099/) (2023)
- [Tracking Any Point with Frame-Event Fusion Network at High Frame Rate](https://mlsanity.com/paper/2409.11953/) (2024)
- [GIFSplat: Generative Prior-Guided Iterative Feed-Forward 3D Gaussian Splatting from Sparse Views](https://mlsanity.com/paper/2602.22571/) (2026)
- [Lie Flow: Video Dynamic Fields Modeling and Predicting with Lie Algebra as Geometric Physics Principle](https://mlsanity.com/paper/2602.21645/) (2026)
- [FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow](https://mlsanity.com/paper/2603.28759/) (2026)

## Abstract (arXiv)

The occlusion problem remains a crucial challenge in optical flow estimation (OFE). Despite the recent significant progress brought about by deep learning, most existing deep learning OFE methods still struggle to handle occlusions; in particular, those based on two frames cannot correctly handle occlusions because occluded regions have no visual correspondences. However, there is still hope in multi-frame settings, which can potentially mitigate the occlusion issue in OFE. Unfortunately, multi-frame OFE (MOFE) remains underexplored, and the limited studies on it are mainly specially designed for pyramid backbones or else obtain the aligned previous frame's features, such as correlation volume and optical flow, through time-consuming backward flow calculation or non-differentiable forward warping transformation. This study proposes an efficient MOFE framework named SplatFlow to address these shortcomings. SplatFlow introduces the differentiable splatting transformation to align the previous frame's motion feature and designs a Final-to-All embedding method to input the aligned motion feature into the current frame's estimation, thus remodeling the existing two-frame backbones. The proposed SplatFlow is efficient yet more accurate, as it can handle occlusions properly. Extensive experimental evaluations show that SplatFlow substantially outperforms all published methods on the KITTI2015 and Sintel benchmarks. Especially on the Sintel benchmark, SplatFlow achieves errors of 1.12 (clean pass) and 2.07 (final pass), with surprisingly significant 19.4% and 16.2% error reductions, respectively, from the previous best results submitted. The code for SplatFlow is available at https://github.com/wwsource/SplatFlow.