# Action-Based Representation Learning for Autonomous Driving

arXiv: https://arxiv.org/abs/2008.09417 · HTML: https://mlsanity.com/paper/2008.09417/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 11 of 11)

- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients](https://mlsanity.com/paper/2204.13683/) (2022)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [Self-Supervised Action-Space Prediction for Automated Driving](https://mlsanity.com/paper/2109.10024/) (2021)
- [Resolving uncertainty on the fly: Modeling adaptive driving behavior as active inference](https://mlsanity.com/paper/2311.06417/) (2023)
- [Perceptual Motor Learning with Active Inference Framework for Robust Lateral Control](https://mlsanity.com/paper/2503.01676/) (2025)
- [Towards Human-Like Driving: Active Inference in Autonomous Vehicle Control](https://mlsanity.com/paper/2407.07684/) (2024)
- [What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planning](https://mlsanity.com/paper/2606.31106/) (2026)
- [Learning to Explain While Planning: Rule-Aligned Diffusion Planning for Autonomous Driving](https://mlsanity.com/paper/2609.39995/) (2026)

## Abstract (arXiv)

Human drivers produce a vast amount of data which could, in principle, be used to improve autonomous driving systems. Unfortunately, seemingly straightforward approaches for creating end-to-end driving models that map sensor data directly into driving actions are problematic in terms of interpretability, and typically have significant difficulty dealing with spurious correlations. Alternatively, we propose to use this kind of action-based driving data for learning representations. Our experiments show that an affordance-based driving model pre-trained with this approach can leverage a relatively small amount of weakly annotated imagery and outperform pure end-to-end driving models, while being more interpretable. Further, we demonstrate how this strategy outperforms previous methods based on learning inverse dynamics models as well as other methods based on heavy human supervision (ImageNet).