mlsanityData to 7 Oct 2026

Multi-task Learning with Attention for End-to-end Autonomous Driving

arXiv 2104.10753 ↗First posted 21 Apr 2021cs.RO cs.AI cs.CVcited by 49

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

also in 1 more
Abstract (arXiv)

Autonomous driving systems need to handle complex scenarios such as lane following, avoiding collisions, taking turns, and responding to traffic signals. In recent years, approaches based on end-to-end behavioral cloning have demonstrated remarkable performance in point-to-point navigational scenarios, using a realistic simulator and standard benchmarks. Offline imitation learning is readily available, as it does not require expensive hand annotation or interaction with the target environment, but it is difficult to obtain a reliable system. In addition, existing methods have not specifically addressed the learning of reaction for traffic lights, which are a rare occurrence in the training datasets. Inspired by the previous work on multi-task learning and attention modeling, we propose a novel multi-task attention-aware network in the conditional imitation learning (CIL) framework. This does not only improve the success rate of standard benchmarks, but also the ability to react to traffic lights, which we show with standard benchmarks.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 26 of its 33 references that are in the map. First posted on arXiv 21 Apr 2021.

Builds on

26 of 33 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
14 more
  1. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  2. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  3. 2017Residual Attention Network for Image Classificationcited by 385
  4. 2019Learning by Cheatingcited by 361
  5. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  6. 2019Few-Shot Object Detection with Attention-RPN and Multi-Relation Detectorcited by 206
  7. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  8. 2018Attention Branch Network: Learning of Attention Mechanism for Visual Explanationcited by 108
  9. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  10. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  11. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  12. 2019Processing Megapixel Images with Deep Attention-Sampling Modelscited by 29
  13. 2020Explaining Autonomous Driving by Learning End-to-End Visual Attentioncited by 25
  14. 2018Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Abilitycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 11; 2024: 12; 2025: 13; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  3. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  4. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  5. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  6. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  7. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  8. 2024HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Constructioncited by 48
  9. 2024Fair Resource Allocation in Multi-Task Learningcited by 34
  10. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  11. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  12. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
37 more
  1. 2022End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agentcited by 21
  2. 2024Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gapcited by 20
  3. 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Drivingcited by 16
  4. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  5. 2025MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perceptioncited by 10
  6. 2023Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Drivingcited by 8
  7. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  8. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  9. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  10. 2021Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environmentscited by 5
  11. 2023AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learningcited by 5
  12. 2024Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Surveycited by 5
  13. 2022Multi-Task Conditional Imitation Learning for Autonomous Navigation at Crowded Intersectionscited by 4
  14. 2025Interpretable Decision-Making for End-to-End Autonomous Drivingcited by 4
  15. 2022DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environmentscited by 3
  16. 2023Sensor Fusion by Spatial Encoding for Autonomous Drivingcited by 3
  17. 2023Addressing Limitations of State-Aware Imitation Learning for Autonomous Drivingcited by 3
  18. 2025A Multi-Task Learning Approach to Linear Multivariate Forecastingcited by 3
  19. 2025Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenterscited by 3
  20. 2025TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Drivingcited by 3
  21. 2022Compressed Hierarchical Representations for Multi-Task Learning and Task Clusteringcited by 2
  22. 2022On the Choice of Data for Efficient Training and Validation of End-to-End Driving Modelscited by 2
  23. 2023DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehiclecited by 2
  24. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  25. 2025RMT-PPAD: Real-time Multi-task Learning for Panoptic Perception in Autonomous Drivingcited by 2
  26. 2026UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perceptioncited by 2
  27. 2022DiRaC-I: Identifying Diverse and Rare Training Classes for Zero-Shot Learningcited by 1
  28. 2023LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learningcited by 1
  29. 2024Guiding Attention in End-to-End Driving Modelscited by 1
  30. 2024Learning Representation for Multitask learning through Self Supervised Auxiliary learningcited by 1
  31. 2025Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigationcited by 1
  32. 2025Advancing Autonomous Driving System Testing: Demands, Challenges, and Future Directionscited by 1
  33. 2025A Survey on Deep Multi-Task Learning in Connected Autonomous Vehiclescited by 0
  34. 2026Cognitive-Causal Multi-Task Learning with Psychological State Conditioning for Assistive Driving Perceptioncited by 0
  35. 2026PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specializationcited by 0
  36. 2026CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learningcited by 0
  37. 2026Revolutionizing Turn-by-Turn Navigation with Cloud-Edge Deep Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Urban Driving with Conditional Imitation Learning
  2. 2024End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning
  3. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization
  4. 2022Dynamic Conditional Imitation Learning for Autonomous Driving
  5. 2023Scaling Vision-based End-to-End Driving with Multi-View Attention Learning
  6. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Driving
  7. 2023Target-point Attention Transformer: A novel trajectory predict network for end-to-end autonomous driving
  8. 2021DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks