# UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model

arXiv: https://arxiv.org/abs/2405.02608 · HTML: https://mlsanity.com/paper/2405.02608/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 76 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 30 of 50)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [Anything-3D: Towards Single-view Anything Reconstruction in the Wild](https://mlsanity.com/paper/2304.10261/) (2023)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)

## Built on by (most cited first, 8 of 8)

- [Flow-Anything: Learning Real-World Optical Flow Estimation from Large-Scale Single-view Images](https://mlsanity.com/paper/2506.07740/) (2025)
- [Towards Dependable Optical Flow: Benchmarks for Real-World Spatiotemporal Corruptions](https://mlsanity.com/paper/2411.14865/) (2024)
- [Rethinking Unsupervised Cross-modal Flow Estimation: Learning from Decoupled Optimization and Consistency Constraint](https://mlsanity.com/paper/2509.24423/) (2025)
- [Adaptive H&E-IHC information fusion staining framework based on feature extra](https://mlsanity.com/paper/2502.20156/) (2025)
- [Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images](https://mlsanity.com/paper/2505.08178/) (2025)
- [GenOpticalFlow: A Generative Approach to Unsupervised Optical Flow Learning](https://mlsanity.com/paper/2603.22270/) (2026)
- [SBF: An Effective Representation to Augment Skeleton for Video-based Human Action Recognition](https://mlsanity.com/paper/2604.03590/) (2026)
- [U2Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2604.10056/) (2026)

## Abstract (arXiv)

Traditional unsupervised optical flow methods are vulnerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAMFlow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further aggregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently.