# Optical Flow for Autonomous Driving: Applications, Challenges and Improvements

arXiv: https://arxiv.org/abs/2301.04422 · HTML: https://mlsanity.com/paper/2301.04422/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Spatial As Deep: Spatial CNN for Traffic Scene Understanding](https://mlsanity.com/paper/1712.06080/) (2017)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving](https://mlsanity.com/paper/1905.01489/) (2019)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving](https://mlsanity.com/paper/2102.07448/) (2021)
- [Generalized Object Detection on Fisheye Cameras for Autonomous Driving: Dataset, Representations and Baseline](https://mlsanity.com/paper/2012.02124/) (2020)
- [FisheyeDistanceNet: Self-Supervised Scale-Aware Distance Estimation using Monocular Fisheye Camera for Autonomous Driving](https://mlsanity.com/paper/1910.04076/) (2019)
- [Milking CowMask for Semi-Supervised Image Classification](https://mlsanity.com/paper/2003.12022/) (2020)
- [Dynamic Task Weighting Methods for Multi-task Networks in Autonomous Driving Systems](https://mlsanity.com/paper/2001.02223/) (2020)
- [Surround-view Fisheye Camera Perception for Automated Driving: Overview, Survey and Challenges](https://mlsanity.com/paper/2205.13281/) (2022)
- [SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for Autonomous Driving](https://mlsanity.com/paper/2203.05056/) (2022)
- [FuseMODNet: Real-Time Camera and LiDAR based Moving Object Detection for robust low-light Autonomous Driving](https://mlsanity.com/paper/1910.05395/) (2019)
- [SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras](https://mlsanity.com/paper/2104.04420/) (2021)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [Imposing Consistency for Optical Flow Estimation](https://mlsanity.com/paper/2204.07262/) (2022)
- [Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline](https://mlsanity.com/paper/2008.07008/) (2020)

## Built on by (most cited first, 9 of 9)

- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2303.06209/) (2023)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challenges](https://mlsanity.com/paper/2402.12041/) (2024)
- [DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning](https://mlsanity.com/paper/2404.06352/) (2024)
- [FisheyeDetNet: 360deg Surround view Fisheye Camera based Object Detection System for Autonomous Driving](https://mlsanity.com/paper/2404.13443/) (2024)
- [Neural Rendering based Urban Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2402.06826/) (2024)
- [U2Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2604.10056/) (2026)
- [FlowPainter: Inpainting Optical Flow via Confidence-Guided Completion](https://mlsanity.com/paper/2607.10140/) (2026)

## Abstract (arXiv)

Optical flow estimation is a well-studied topic for automated driving applications. Many outstanding optical flow estimation methods have been proposed, but they become erroneous when tested in challenging scenarios that are commonly encountered. Despite the increasing use of fisheye cameras for near-field sensing in automated driving, there is very limited literature on optical flow estimation with strong lens distortion. Thus we propose and evaluate training strategies to improve a learning-based optical flow algorithm by leveraging the only existing fisheye dataset with optical flow ground truth. While trained with synthetic data, the model demonstrates strong capabilities to generalize to real world fisheye data. The other challenge neglected by existing state-of-the-art algorithms is low light. We propose a novel, generic semi-supervised framework that significantly boosts performances of existing methods in such conditions. To the best of our knowledge, this is the first approach that explicitly handles optical flow estimation in low light.