# SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model

arXiv: https://arxiv.org/abs/2307.16586 · HTML: https://mlsanity.com/paper/2307.16586/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://mlsanity.com/paper/1812.06264/) (2018)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization](https://mlsanity.com/paper/1903.07414/) (2019)
- [Global Matching with Overlapping Attention for Optical Flow Estimation](https://mlsanity.com/paper/2203.11335/) (2022)
- [SKFlow: Learning Optical Flow with Super Kernels](https://mlsanity.com/paper/2205.14623/) (2022)
- [Learning Optical Flow from a Few Matches](https://mlsanity.com/paper/2104.02166/) (2021)

## Built on by (most cited first, 15 of 15)

- [WAFT: Warping-Alone Field Transforms for Optical Flow](https://mlsanity.com/paper/2506.21526/) (2025)
- [DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework](https://mlsanity.com/paper/2503.14880/) (2025)
- [StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences](https://mlsanity.com/paper/2311.17099/) (2023)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model](https://mlsanity.com/paper/2405.02608/) (2024)
- [FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model](https://mlsanity.com/paper/2409.05396/) (2024)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [MegaFlow: Zero-Shot Large Displacement Optical Flow](https://mlsanity.com/paper/2603.25739/) (2026)
- [Towards Dependable Optical Flow: Benchmarks for Real-World Spatiotemporal Corruptions](https://mlsanity.com/paper/2411.14865/) (2024)
- [PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models](https://mlsanity.com/paper/2507.08400/) (2025)
- [Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation](https://mlsanity.com/paper/2312.07180/) (2023)
- [Adaptive H&E-IHC information fusion staining framework based on feature extra](https://mlsanity.com/paper/2502.20156/) (2025)
- [Flow-CDNet: A Novel Network for Detecting Both Slow and Fast Changes in Bitemporal Images](https://mlsanity.com/paper/2507.02307/) (2025)
- [FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow](https://mlsanity.com/paper/2603.28759/) (2026)
- [FAST: Flow Any Scene Transformer](https://mlsanity.com/paper/2609.39748/) (2026)

## Abstract (arXiv)

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.