mlsanityData to 13 Oct 2026

Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving

arXiv 2310.20650 ↗First posted 31 Oct 2023cs.CV cs.ROcited by 3

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

also in 1 more
Abstract (arXiv)

Conditional Imitation learning is a common and effective approach to train autonomous driving agents. However, two issues limit the full potential of this approach: (i) the inertia problem, a special case of causal confusion where the agent mistakenly correlates low speed with no acceleration, and (ii) low correlation between offline and online performance due to the accumulation of small errors that brings the agent in a previously unseen state. Both issues are critical for state-aware models, yet informing the driving agent of its internal state as well as the state of the environment is of crucial importance. In this paper we propose a multi-task learning agent based on a multi-stage vision transformer with state token propagation. We feed the state of the vehicle along with the representation of the environment as a special token of the transformer and propagate it throughout the network. This allows us to tackle the aforementioned issues from different angles: guiding the driving policy with learned stop/go information, performing data augmentation directly on the state of the vehicle and visually explaining the model's decisions. We report a drastic decrease in inertia and a high correlation between offline and online metrics.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 36 of its 53 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

36 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Learning from All Vehicles22 Mar
  2. 2021Causal Imitative Model for Autonomous Driving7 Dec
  3. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coach18 Aug
  4. 2021Multi-task Learning with Attention for End-to-end Autonomous Driving21 Apr

Other references · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2017Billion-scale similarity search with GPUscited by 2,490
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,439
  5. 2016End to End Learning for Self-Driving Carscited by 1,002
  6. 2017End-to-end Driving via Conditional Imitation Learningcited by 480
  7. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 479
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  9. 2019Learning by Cheatingcited by 361
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2019Causal Confusion in Imitation Learningcited by 287
  12. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 231
20 more
  1. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  2. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 149
  3. 2021Learning to drive from a world on railscited by 114
  4. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  5. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  6. 2021Explanations in Autonomous Driving: A Surveycited by 81
  7. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  8. 2016Full-Resolution Residual Networks for Semantic Segmentation in Street Scenescited by 71
  9. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  10. 2018On Offline Evaluation of Vision-based Driving Modelscited by 64
  11. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 60
  12. 2018Context-Aware Synthesis and Placement of Object Instancescited by 41
  13. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  14. 2018End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptioncited by 27
  15. 2020Explaining Autonomous Driving by Learning End-to-End Visual Attentioncited by 25
  16. 2018Global overview of Imitation Learningcited by 16
  17. 2022CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Drivingcited by 16
  18. 2022Resolving Copycat Problems in Visual Imitation Learning via Residual Action Predictioncited by 14
  19. 2018Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Abilitycited by 12
  20. 2020Multiple Future Prediction Leveraging Synthetic Trajectoriescited by 5

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  2. 2024PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehiclescited by 3
  3. 2024Behavioral Cloning Models Reality Check for Autonomous Drivingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2019Safer End-to-End Autonomous Driving via Conditional Imitation Learning and Command Augmentation
  2. 2019Urban Driving with Conditional Imitation Learning
  3. 2025State-Conditional Adversarial Learning: An Off-Policy Visual Domain Transfer Method for End-to-End Imitation Learning
  4. 2025From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
  5. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization
  6. 2025Sequence of Expert: Boosting Imitation Planners for Autonomous Driving through Temporal Alternation
  7. 2022Model-Based Imitation Learning for Urban Driving
  8. 2019Conditional Vehicle Trajectories Prediction in CARLA Urban Environment