# Visuomotor Understanding for Representation Learning of Driving Scenes

arXiv: https://arxiv.org/abs/1909.06979 · HTML: https://mlsanity.com/paper/1909.06979/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency](https://mlsanity.com/paper/2102.02629/) (2021)
- [SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations](https://mlsanity.com/paper/2112.04680/) (2021)
- [Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation](https://mlsanity.com/paper/2108.05773/) (2021)
- [Universal Adversarial Perturbations Through the Lens of Deep Steganography: Towards A Fourier Perspective](https://mlsanity.com/paper/2102.06479/) (2021)
- [Optical Flow Estimation from a Single Motion-blurred Image](https://mlsanity.com/paper/2103.02996/) (2021)

## Abstract (arXiv)

Dashboard cameras capture a tremendous amount of driving scene video each day. These videos are purposefully coupled with vehicle sensing data, such as from the speedometer and inertial sensors, providing an additional sensing modality for free. In this work, we leverage the large-scale unlabeled yet naturally paired data for visual representation learning in the driving scenario. A representation is learned in an end-to-end self-supervised framework for predicting dense optical flow from a single frame with paired sensing data. We postulate that success on this task requires the network to learn semantic and geometric knowledge in the ego-centric view. For example, forecasting a future view to be seen from a moving vehicle requires an understanding of scene depth, scale, and movement of objects. We demonstrate that our learned representation can benefit other tasks that require detailed scene understanding and outperforms competing unsupervised representations on semantic segmentation.