# Interpretable End-to-End Driving Model for Implicit Scene Understanding

arXiv: https://arxiv.org/abs/2308.01180 · HTML: https://mlsanity.com/paper/2308.01180/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 30 of its 31 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 30 of 30)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [Learning to drive from a world on rails](https://mlsanity.com/paper/2105.00636/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Detecting Visual Relationships with Deep Relational Networks](https://mlsanity.com/paper/1704.03114/) (2017)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles](https://mlsanity.com/paper/2104.07830/) (2021)
- [Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction](https://mlsanity.com/paper/2111.06123/) (2021)
- [Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving](https://mlsanity.com/paper/2011.01153/) (2020)
- [RSG-Net: Towards Rich Sematic Relationship Prediction for Intelligent Vehicle in Complex Environments](https://mlsanity.com/paper/2207.12321/) (2022)

## Built on by (most cited first, 2 of 2)

- [ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts](https://mlsanity.com/paper/2511.11740/) (2025)
- [Argus: Resilience-Oriented Safety Assurance Framework for End-to-End ADSs](https://mlsanity.com/paper/2511.09032/) (2025)

## Abstract (arXiv)

Driving scene understanding is to obtain comprehensive scene information through the sensor data and provide a basis for downstream tasks, which is indispensable for the safety of self-driving vehicles. Specific perception tasks, such as object detection and scene graph generation, are commonly used. However, the results of these tasks are only equivalent to the characterization of sampling from high-dimensional scene features, which are not sufficient to represent the scenario. In addition, the goal of perception tasks is inconsistent with human driving that just focuses on what may affect the ego-trajectory. Therefore, we propose an end-to-end Interpretable Implicit Driving Scene Understanding (II-DSU) model to extract implicit high-dimensional scene features as scene understanding results guided by a planning module and to validate the plausibility of scene understanding using auxiliary perception tasks for visualization. Experimental results on CARLA benchmarks show that our approach achieves the new state-of-the-art and is able to obtain scene features that embody richer scene information relevant to driving, enabling superior performance of the downstream planning.