# Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting

arXiv: https://arxiv.org/abs/2107.12674 · HTML: https://mlsanity.com/paper/2107.12674/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 23 of 23)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Deep Factors for Forecasting](https://mlsanity.com/paper/1905.12417/) (2019)
- [Multi-Fiber Networks for Video Recognition](https://mlsanity.com/paper/1807.11195/) (2018)
- [A Commute in Data: The comma2k19 Dataset](https://mlsanity.com/paper/1812.05752/) (2018)
- [End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception](https://mlsanity.com/paper/1801.06734/) (2018)
- [Neural Machine Translation and Sequence-to-sequence Models: A Tutorial](https://mlsanity.com/paper/1703.01619/) (2017)
- [Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks](https://mlsanity.com/paper/1811.02759/) (2018)
- [A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks](https://mlsanity.com/paper/2007.00800/) (2020)
- [Self-Driving Car Steering Angle Prediction Based on Image Recognition](https://mlsanity.com/paper/1912.05440/) (2019)
- [Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction](https://mlsanity.com/paper/1901.08096/) (2019)
- [Feudal Steering: Hierarchical Learning for Steering Angle Prediction](https://mlsanity.com/paper/2006.06869/) (2020)

## Built on by (most cited first, 1 of 1)

- [STRIDE: Street View-based Environmental Feature Detection and Pedestrian Collision Prediction](https://mlsanity.com/paper/2308.13183/) (2023)

## Abstract (arXiv)

Autonomous driving gained huge traction in recent years, due to its potential to change the way we commute. Much effort has been put into trying to estimate the state of a vehicle. Meanwhile, learning to forecast the state of a vehicle ahead introduces new capabilities, such as predicting dangerous situations. Moreover, forecasting brings new supervision opportunities by learning to predict richer a context, expressed by multiple horizons. Intuitively, a video stream originated from a front-facing camera is necessary because it encodes information about the upcoming road. Besides, historical traces of the vehicle's states give more context. In this paper, we tackle multi-horizon forecasting of vehicle states by fusing the two modalities. We design and experiment with 3 end-to-end architectures that exploit 3D convolutions for visual features extraction and 1D convolutions for features extraction from speed and steering angle traces. To demonstrate the effectiveness of our method, we perform extensive experiments on two publicly available real-world datasets, Comma2k19 and the Udacity challenge. We show that we are able to forecast a vehicle's state to various horizons, while outperforming the current state-of-the-art results on the related task of driving state estimation. We examine the contribution of vision features, and find that a model fed with vision features achieves an error that is 56.6% and 66.9% of the error of a model that doesn't use those features, on the Udacity and Comma2k19 datasets respectively.