mlsanityData to 7 Oct 2026

A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directions

arXiv 2403.07542 ↗First posted 12 Mar 2024cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

This survey explores the adaptation of visual transformer models in Autonomous Driving, a transition inspired by their success in Natural Language Processing. Surpassing traditional Recurrent Neural Networks in tasks like sequential image processing and outperforming Convolutional Neural Networks in global context capture, as evidenced in complex scene recognition, Transformers are gaining traction in computer vision. These capabilities are crucial in Autonomous Driving for real-time, dynamic visual scene processing. Our survey provides a comprehensive overview of Vision Transformer applications in Autonomous Driving, focusing on foundational concepts such as self-attention, multi-head attention, and encoder-decoder architecture. We cover applications in object detection, segmentation, pedestrian detection, lane detection, and more, comparing their architectural merits and limitations. The survey concludes with future research directions, highlighting the growing role of Vision Transformers in Autonomous Driving.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2022Planning-oriented Autonomous Drivingcited by 1,068
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
38 more
  1. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  2. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  3. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  4. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  5. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  6. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  7. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  8. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  9. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  10. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  11. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  12. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 311
  13. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  14. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  15. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  16. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  17. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  18. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  19. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  20. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  21. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  22. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  23. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  24. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  25. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  26. 2022BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigscited by 123
  27. 2021Translating Images into Mapscited by 115
  28. 2020End-to-end Lane Shape Prediction with Transformerscited by 111
  29. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  30. 2021Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Imagescited by 98
  31. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  32. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  33. 2023RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Drivingcited by 76
  34. 2022Future Transformer for Long-term Action Anticipationcited by 57
  35. 2022CurveFormer: 3D Lane Detection by Curve Propagation with Curve Queries and Attentioncited by 40
  36. 2022Laneformer: Object-aware Row-Column Transformers for Lane Detectioncited by 25
  37. 2022Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Drivingcited by 24
  38. 2020Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odysseycited by 21

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Towards Interpretable and Efficient Attention: Compressing All by Contracting a Fewcited by 2
  2. 2024Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformerscited by 1
  3. 2025A Novel Vision Transformer for Camera-LiDAR Fusion based Traffic Object Segmentationcited by 1
  4. 2025Elytra: A Flexible Framework for Securing Large Vision Systemscited by 0