# Joint Optical Flow and Temporally Consistent Semantic Segmentation

arXiv: https://arxiv.org/abs/1607.07716 · HTML: https://mlsanity.com/paper/1607.07716/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2016.

## Built on by (most cited first, 19 of 19)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Multi-scale Matching Networks for Semantic Correspondence](https://mlsanity.com/paper/2108.00211/) (2021)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [Perceptual Consistency in Video Segmentation](https://mlsanity.com/paper/2110.12385/) (2021)
- [RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation](https://mlsanity.com/paper/2203.04187/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Pseudo Supervision](https://mlsanity.com/paper/2207.02372/) (2022)
- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2303.06209/) (2023)
- [UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model](https://mlsanity.com/paper/2405.02608/) (2024)
- [Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation](https://mlsanity.com/paper/2309.08020/) (2023)
- [Detail Preserving Residual Feature Pyramid Modules for Optical Flow](https://mlsanity.com/paper/2107.10990/) (2021)
- [What can human minimal videos tell us about dynamic recognition models?](https://mlsanity.com/paper/2104.09447/) (2021)
- [OAS-Net: Occlusion Aware Sampling Network for Accurate Optical Flow](https://mlsanity.com/paper/2102.00364/) (2021)
- [MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion](https://mlsanity.com/paper/2405.19921/) (2024)
- [Deep Learning Method for Cell-Wise Object Tracking, Velocity Estimation and Projection of Sensor Data over Time](https://mlsanity.com/paper/2306.06126/) (2023)
- [Motion-Corrected Moving Average: Including Post-Hoc Temporal Information for Improved Video Segmentation](https://mlsanity.com/paper/2403.03120/) (2024)
- [Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving](https://mlsanity.com/paper/2602.10052/) (2026)

## Abstract (arXiv)

The importance and demands of visual scene understanding have been steadily increasing along with the active development of autonomous systems. Consequently, there has been a large amount of research dedicated to semantic segmentation and dense motion estimation. In this paper, we propose a method for jointly estimating optical flow and temporally consistent semantic segmentation, which closely connects these two problem domains and leverages each other. Semantic segmentation provides information on plausible physical motion to its associated pixels, and accurate pixel-level temporal correspondences enhance the accuracy of semantic segmentation in the temporal domain. We demonstrate the benefits of our approach on the KITTI benchmark, where we observe performance gains for flow and segmentation. We achieve state-of-the-art optical flow results, and outperform all published algorithms by a large margin on challenging, but crucial dynamic objects.