mlsanityData to 7 Oct 2026

NEAT: Neural Attention Fields for End-to-End Autonomous Driving

arXiv 2109.04456 ↗First posted 9 Sep 2021cs.CV cs.AI cs.LGcited by 174

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Efficient reasoning about the semantic, spatial, and temporal structure of a scene is a crucial prerequisite for autonomous driving. We present NEural ATtention fields (NEAT), a novel representation that enables such reasoning for end-to-end imitation learning models. NEAT is a continuous function which maps locations in Bird's Eye View (BEV) scene coordinates to waypoints and semantics, using intermediate attention maps to iteratively compress high-dimensional 2D image features into a compact representation. This allows our model to selectively attend to relevant regions in the input while ignoring information irrelevant to the driving task, effectively associating the images with the BEV representation. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert used to generate its training data. Furthermore, visualizing the attention maps for models with NEAT intermediate representations provides improved interpretability.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 69 of its 81 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

69 of 81 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  12. 2020Convolutional Occupancy Networkscited by 760
38 more
  1. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  2. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  3. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  4. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  7. 2020Local Implicit Grid Representations for 3D Scenescited by 372
  8. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  9. 2019Learning by Cheatingcited by 361
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  12. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  13. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  14. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  15. 2019Learning Shape Templates with Structured Implicit Functionscited by 242
  16. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  17. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  18. 2016Adversarially Learned Inferencecited by 216
  19. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  20. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  21. 2019Texture Fields: Learning Texture Representations in Function Spacecited by 210
  22. 2019BSP-Net: Generating Compact Meshes via Binary Space Partitioningcited by 205
  23. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  24. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
  25. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  26. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 173
  27. 2019CvxNet: Learnable Convex Decompositioncited by 173
  28. 2017Modulating early visual processing by languagecited by 167
  29. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  30. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  31. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  32. 2020A Survey of End-to-End Driving: Architectures and Training Methodscited by 123
  33. 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 116
  34. 2021Learning to drive from a world on railscited by 114
  35. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  36. 2019Learning to Infer Implicit Surfaces without 3D Supervisioncited by 103
  37. 2018Driving Policy Transfer via Modularity and Abstractioncited by 91
  38. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 35; 2023: 46; 2024: 45; 2025: 31; 2026: 15.

174 papers.

Sort:

By area:

  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2021Neural Fields in Visual Computing and Beyondcited by 539
  4. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  6. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  7. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  8. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  9. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  10. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  11. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  12. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
38 more
  1. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  2. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 155
  3. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  4. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  5. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  6. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  7. 2022Learning from All Vehiclescited by 135
  8. 2022BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigscited by 123
  9. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  10. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  11. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  12. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  13. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  14. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  15. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 95
  16. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  17. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  18. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  19. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  20. 2023Bayes' Rays: Uncertainty Quantification for Neural Radiance Fieldscited by 90
  21. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  22. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  23. 2023End-to-End Vectorized HD-map Construction with Piecewise Bezier Curvecited by 80
  24. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  25. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  26. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  27. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 66
  28. 2023Nearest Neighbor Guidance for Out-of-Distribution Detectioncited by 64
  29. 2025A Survey of World Models for Autonomous Drivingcited by 60
  30. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  31. 2022MVSTER: Epipolar Transformer for Efficient Multi-View Stereocited by 50
  32. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  33. 2022Learning Ego 3D Representation as Ray Tracingcited by 43
  34. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41
  35. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  36. 2022X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentationcited by 38
  37. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  38. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
  2. 2025Vision-Language Cross-Attention for Real-Time Autonomous Driving
  3. 2026FocusDrive: Reasoning with Visual Focus for Autonomous Driving
  4. 2025FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling
  5. 2019Utilizing Eye Gaze to Enhance the Generalization of Imitation Networks to Unseen Environments
  6. 2025Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning
  7. 2021Multi-task Learning with Attention for End-to-end Autonomous Driving
  8. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations