mlsanityData to 7 Oct 2026

Embodied Understanding of Driving Scenarios

arXiv 2403.04593 ↗First posted 7 Mar 2024cs.CVcited by 63

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 79 of its 98 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

79 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  3. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving11 Sep
  4. 2023Language Prompt for Autonomous Driving8 Sep

Other references · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  13. 2020Object-Centric Learning with Slot Attentioncited by 865
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  15. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  18. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  19. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  20. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  24. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  25. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  26. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  27. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  28. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  29. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  30. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  31. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  32. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  33. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  34. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  35. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  36. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  37. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  38. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 31; 2026: 15.

63 papers.

Sort:

By area:

  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  4. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  5. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  6. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  7. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  8. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  9. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  10. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  11. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  12. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
38 more
  1. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  2. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  3. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  4. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  5. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  6. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  7. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  8. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  9. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  10. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  11. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  12. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  13. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  14. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinkingcited by 13
  15. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  16. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 11
  17. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  18. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  19. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  20. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  21. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  22. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Drivingcited by 5
  23. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  24. 2026Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Drivingcited by 4
  25. 2025TimeLoc: A Unified End-to-End Framework for Precise Timestamp Localization in Long Videoscited by 3
  26. 2025Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Drivingcited by 3
  27. 2025iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoningcited by 3
  28. 2025Universal Camouflage Attack on Vision-Language Models for Autonomous Drivingcited by 3
  29. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  30. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  31. 2026EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillationcited by 3
  32. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  33. 2026VLGA: Vision-Language-Geometry-Action Models for Autonomous Drivingcited by 3
  34. 2024A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directionscited by 2
  35. 2024How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perceptioncited by 2
  36. 2024METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidancecited by 2
  37. 2025TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videoscited by 2
  38. 2025OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Drivingcited by 2

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  3. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving
  4. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
  5. 2024Probing Multimodal LLMs as World Models for Driving
  6. 2025From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving
  7. 2025DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models
  8. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities