# Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering

arXiv: https://arxiv.org/abs/2409.12716 · HTML: https://mlsanity.com/paper/2409.12716/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention](https://mlsanity.com/paper/1703.10631/) (2017)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [Computing Systems for Autonomous Driving: State-of-the-Art and Challenges](https://mlsanity.com/paper/2009.14349/) (2020)
- [Emergent Visual Sensors for Autonomous Vehicles](https://mlsanity.com/paper/2205.09383/) (2022)
- [End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies](https://mlsanity.com/paper/1710.03804/) (2017)
- [Optical Flow based Visual Potential Field for Autonomous Driving](https://mlsanity.com/paper/2006.00168/) (2020)
- [Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images](https://mlsanity.com/paper/1912.11691/) (2019)
- [End-to-end Autonomous Driving using Deep Learning: A Systematic Review](https://mlsanity.com/paper/2311.18636/) (2023)
- [Exploring Latent Pathways: Enhancing the Interpretability of Autonomous Driving with a Variational Autoencoder](https://mlsanity.com/paper/2404.01750/) (2024)
- [Multimodal End-to-End Learning for Autonomous Steering in Adverse Road and Weather Conditions](https://mlsanity.com/paper/2010.14924/) (2020)
- [Lateral control for autonomous vehicles: A comparative evaluation](https://mlsanity.com/paper/2311.07987/) (2023)
- [Multi-Modal Fusion for Sensorimotor Coordination in Steering Angle Prediction](https://mlsanity.com/paper/2202.05500/) (2022)
- [Why Autonomous Vehicles Are Not Ready Yet: A Multi-Disciplinary Review of Problems, Attempted Solutions, and Future Directions](https://mlsanity.com/paper/2311.09093/) (2023)

## Built on by (most cited first, 2 of 2)

- [Depth Matters: Multimodal RGB-D Perception for Robust Autonomous Agents](https://mlsanity.com/paper/2503.16711/) (2025)
- [Enhancing Steering Estimation with Semantic-Aware GNNs](https://mlsanity.com/paper/2503.17153/) (2025)

## Abstract (arXiv)

Autonomous vehicle navigation is a key challenge in artificial intelligence, requiring robust and accurate decision-making processes. This research introduces a new end-to-end method that exploits multimodal information from a single monocular camera to improve the steering predictions for self-driving cars. Unlike conventional models that require several sensors which can be costly and complex or rely exclusively on RGB images that may not be robust enough under different conditions, our model significantly improves vehicle steering prediction performance from a single visual sensor. By focusing on the fusion of RGB imagery with depth completion information or optical flow data, we propose a comprehensive framework that integrates these modalities through both early and hybrid fusion techniques. We use three distinct neural network models to implement our approach: Convolution Neural Network - Neutral Circuit Policy (CNN-NCP) , Variational Auto Encoder - Long Short-Term Memory (VAE-LSTM) , and Neural Circuit Policy architecture VAE-NCP. By incorporating optical flow into the decision-making process, our method significantly advances autonomous navigation. Empirical results from our comparative study using Boston driving data show that our model, which integrates image and motion information, is robust and reliable. It outperforms state-of-the-art approaches that do not use optical flow, reducing the steering estimation error by 31%. This demonstrates the potential of optical flow data, combined with advanced neural network architectures (a CNN-based structure for fusing data and a Recurrence-based network for inferring a command from latent space), to enhance the performance of autonomous vehicles steering estimation.