# Optical Flow with Semantic Segmentation and Localized Layers

arXiv: https://arxiv.org/abs/1603.03911 · HTML: https://mlsanity.com/paper/1603.03911/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2016.

## Built on by (most cited first, 21 of 21)

- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Many-to-many Splatting for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2204.03513/) (2022)
- [Exploiting Temporal State Space Sharing for Video Semantic Segmentation](https://mlsanity.com/paper/2503.20824/) (2025)
- [SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2303.06209/) (2023)
- [ASFlow: Unsupervised Optical Flow Learning with Adaptive Pyramid Sampling](https://mlsanity.com/paper/2104.03560/) (2021)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model](https://mlsanity.com/paper/2405.02608/) (2024)
- [GOSS: Towards Generalized Open-set Semantic Segmentation](https://mlsanity.com/paper/2203.12116/) (2022)
- [Detail Preserving Residual Feature Pyramid Modules for Optical Flow](https://mlsanity.com/paper/2107.10990/) (2021)
- [Zero-Parameter Geometric Gating for Temporally Stable Low-Altitude UAV Video Semantic Segmentation](https://mlsanity.com/paper/2606.09162/) (2026)
- [Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement](https://mlsanity.com/paper/2310.18946/) (2023)
- [Spatio-Temporal driven Attention Graph Neural Network with Block Adjacency matrix (STAG-NN-BA) for Remote Land-use Change Detection](https://mlsanity.com/paper/2303.14322/) (2023)
- [Log-Polar Space Convolution for Convolutional Neural Networks](https://mlsanity.com/paper/2107.11943/) (2021)
- [Learning To Segment Dominant Object Motion From Watching Videos](https://mlsanity.com/paper/2111.14160/) (2021)
- [A Universal Railway Obstacle Detection System based on Semi-supervised Segmentation And Optical Flow](https://mlsanity.com/paper/2406.18908/) (2024)
- [Physics-Guided Motion Loss for Video Generation Model](https://mlsanity.com/paper/2506.02244/) (2025)
- [RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation](https://mlsanity.com/paper/2603.24295/) (2026)
- [Unsupervised Learning of Depth, Camera Pose and Optical Flow from Monocular Video](https://mlsanity.com/paper/2205.09821/) (2022)
- [Leveraging Object Priors for Point Tracking](https://mlsanity.com/paper/2409.05786/) (2024)
- [Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts](https://mlsanity.com/paper/2604.17115/) (2026)
- [Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing](https://mlsanity.com/paper/2607.00124/) (2026)

## Abstract (arXiv)

Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.