mlsanityData to 7 Oct 2026

OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

arXiv 2409.03272 ↗First posted 5 Sep 2024cs.CV cs.ROcited by 82

On the map: World models → World models for autonomous driving

also in 2 more
Abstract (arXiv)

The rise of multi-modal large language models(MLLMs) has spurred their applications in autonomous driving. Recent MLLM-based methods perform action by learning a direct mapping from perception to action, neglecting the dynamics of the world and the relations between action and world dynamics. In contrast, human beings possess world model that enables them to simulate the future states based on 3D internal visual representation and plan actions accordingly. To this end, we propose OccLLaMA, an occupancy-language-action generative world model, which uses semantic occupancy as a general visual representation and unifies vision-language-action(VLA) modalities through an autoregressive model. Specifically, we introduce a novel VQVAE-like scene tokenizer to efficiently discretize and reconstruct semantic occupancy scenes, considering its sparsity and classes imbalance. Then, we build a unified multi-modal vocabulary for vision, language and action. Furthermore, we enhance LLM, specifically LLaMA, to perform the next token/scene prediction on the unified vocabulary to complete multiple tasks in autonomous driving. Extensive experiments demonstrate that OccLLaMA achieves competitive performance across multiple tasks, including 4D occupancy forecasting, motion planning, and visual question answering, showcasing its potential as a foundation model in autonomous driving.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

34 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents8 Jul
  2. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  3. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  4. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  5. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  6. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving27 Apr
  7. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  8. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  10. 2018World Modelscited by 1,444
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  12. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
14 more
  1. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  2. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  5. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 162
  6. 2023A Language Agent for Autonomous Drivingcited by 161
  7. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  8. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  9. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  10. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  11. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  12. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  13. 2023MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersectionscited by 38
  14. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 47; 2026: 26.

82 papers.

Sort:

By area:

  1. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  2. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  5. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  6. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 20253D and 4D World Modeling: A Surveycited by 64
  9. 2025A Survey of World Models for Autonomous Drivingcited by 60
  10. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  11. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  12. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
38 more
  1. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  2. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  3. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  5. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  7. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Drivingcited by 29
  8. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  9. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  10. 2025UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Drivingcited by 27
  11. 20253D Scene Generation: A Surveycited by 24
  12. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  13. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  14. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  15. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  16. 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
  17. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  18. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  19. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervisioncited by 16
  20. 2025When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Modelscited by 16
  21. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  22. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  23. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
  24. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  25. 2025Four Principles for Physically Interpretable World Modelscited by 10
  26. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  27. 2025SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queriescited by 10
  28. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  29. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  30. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Predictioncited by 9
  31. 2026Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Predictioncited by 9
  32. 2025Occupancy World Model for Robotscited by 8
  33. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  34. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
  35. 2026Chain of World: World Model Thinking in Latent Motioncited by 8
  36. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  37. 2026CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuningcited by 7
  38. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models
  2. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
  3. 2026Learning Vision-Language-Action World Models for Autonomous Driving
  4. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  5. 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
  7. 2025VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Driving
  8. 2024OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving