# Optical Flow augmented Semantic Segmentation networks for Automated Driving

arXiv: https://arxiv.org/abs/1901.07355 · HTML: https://mlsanity.com/paper/1901.07355/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2019.

## Built on by (most cited first, 11 of 11)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [Exploring Event-driven Dynamic Context for Accident Scene Segmentation](https://mlsanity.com/paper/2112.05006/) (2021)
- [Sequential vessel segmentation via deep channel attention network](https://mlsanity.com/paper/2102.05229/) (2021)
- [SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2303.06209/) (2023)
- [Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2501.07806/) (2025)
- [RALACs: Action Recognition in Autonomous Vehicles using Interaction Encoding and Optical Flow](https://mlsanity.com/paper/2209.14408/) (2022)
- [DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning](https://mlsanity.com/paper/2404.06352/) (2024)
- [Neural Rendering based Urban Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2402.06826/) (2024)
- [Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical Instrument Segmentation](https://mlsanity.com/paper/2403.10216/) (2024)
- [Impact of Video Compression Artifacts on Fisheye Camera Visual Perception Tasks](https://mlsanity.com/paper/2403.16338/) (2024)

## Abstract (arXiv)

Motion is a dominant cue in automated driving systems. Optical flow is typically computed to detect moving objects and to estimate depth using triangulation. In this paper, our motivation is to leverage the existing dense optical flow to improve the performance of semantic segmentation. To provide a systematic study, we construct four different architectures which use RGB only, flow only, RGBF concatenated and two-stream RGB + flow. We evaluate these networks on two automotive datasets namely Virtual KITTI and Cityscapes using the state-of-the-art flow estimator FlowNet v2. We also make use of the ground truth optical flow in Virtual KITTI to serve as an ideal estimator and a standard Farneback optical flow algorithm to study the effect of noise. Using the flow ground truth in Virtual KITTI, two-stream architecture achieves the best results with an improvement of 4% IoU. As expected, there is a large improvement for moving objects like trucks, vans and cars with 38%, 28% and 6% increase in IoU. FlowNet produces an improvement of 2.4% in average IoU with larger improvement in the moving objects corresponding to 26%, 11% and 5% in trucks, vans and cars. In Cityscapes, flow augmentation provided an improvement for moving objects like motorcycle and train with an increase of 17% and 7% in IoU.