mlsanityData to 7 Oct 2026

Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

arXiv 2407.00959 ↗First posted 1 Jul 2024cs.AI cs.ROcited by 55

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system design. Traditional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM's reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by leveraging a traditional end-to-end driving model to produce condensed and semantically enriched representations of the scene, which are optimized for LLM planning compatibility through deliberate representation and reasoning alignment training stages. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 30 of its 32 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

30 of 32 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2022Planning-oriented Autonomous Drivingcited by 1,068
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  12. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
18 more
  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  3. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  5. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  6. 2023A Language Agent for Autonomous Drivingcited by 161
  7. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  8. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  9. 2022Object Scene Representation Transformercited by 90
  10. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  11. 2023Semantic Anomaly Detection with Large Language Modelscited by 53
  12. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  13. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  14. 2022A System-Level View on Out-of-Distribution Data in Roboticscited by 33
  15. 2021Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trendscited by 26
  16. 2021Safety Assurances for Human-Robot Interaction via Confidence-aware Game-theoretic Human Modelscited by 21
  17. 2021On complementing end-to-end human behavior predictors with planningcited by 10
  18. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 33; 2026: 15.

55 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  5. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  6. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  7. 2025A Survey of World Models for Autonomous Drivingcited by 60
  8. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  10. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  11. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  12. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
38 more
  1. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  2. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  3. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representationcited by 18
  4. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  5. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  6. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 14
  7. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  8. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  9. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  10. 2025Position: Foundation Models Need Digital Twin Representationscited by 10
  11. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  12. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  13. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Drivingcited by 9
  14. 2025SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulationcited by 9
  15. 2025RoCA: Robust Cross-Domain End-to-End Autonomous Drivingcited by 8
  16. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  17. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Drivingcited by 6
  18. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  19. 2025ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Drivingcited by 6
  20. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
  21. 2026Generative Scenario Rollouts for End-to-End Autonomous Drivingcited by 6
  22. 2026Accelerating Structured Chain-of-Thought in Autonomous Vehiclescited by 6
  23. 2024Query3D: LLM-Powered Open-Vocabulary Scene Segmentation with Language Embedded 3D Gaussiancited by 5
  24. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  25. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  26. 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
  27. 2025ZeroVO: Visual Odometry with Minimal Assumptionscited by 4
  28. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  29. 2024MTA: Multimodal Task Alignment for BEV Perception and Captioningcited by 3
  30. 2025Scaling LLM Pre-training with Vocabulary Curriculumcited by 3
  31. 2025Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Drivingcited by 3
  32. 2025SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modelingcited by 3
  33. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  34. 2026ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Drivingcited by 3
  35. 2026RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Timecited by 3
  36. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
  37. 2026OpenLongTail: Generative Scaling of Long-Tail Driving Datacited by 1
  38. 2026Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compressioncited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignment
  2. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  3. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving
  4. 2025Align2Act: Instruction-Tuned Models for Human-Aligned Autonomous Driving
  5. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  6. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  7. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
  8. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving