mlsanityData to 7 Oct 2026

Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion

arXiv 2311.01017 ↗First posted 2 Nov 2023cs.CV cs.AI cs.LGcited by 105

On the map: World models → World models for autonomous driving

also in 2 more
Abstract (arXiv)

Learning world models can teach an agent how the world works in an unsupervised manner. Even though it can be viewed as a special case of sequence modeling, progress for scaling world models on robotic applications such as autonomous driving has been somewhat less rapid than scaling language models with Generative Pre-trained Transformers (GPT). We identify two reasons as major bottlenecks: dealing with complex and unstructured observation space, and having a scalable generative model. Consequently, we propose Copilot4D, a novel world modeling approach that first tokenizes sensor observations with VQVAE, then predicts the future via discrete diffusion. To efficiently decode and denoise tokens in parallel, we recast Masked Generative Image Transformer as discrete diffusion and enhance it with a few simple changes, resulting in notable improvement. When applied to learning world models on point cloud observations, Copilot4D reduces prior SOTA Chamfer distance by more than 65% for 1s prediction, and more than 50% for 3s prediction, across NuScenes, KITTI Odometry, and Argoverse2 datasets. Our results demonstrate that discrete diffusion on tokenized agent experience can unlock the power of GPT-like unsupervised learning for robotics.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 52 of its 64 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

52 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2016Layer Normalizationcited by 5,579
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2021Variational Diffusion Modelscited by 1,145
  13. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  14. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  15. 2020Mastering Atari with Discrete World Modelscited by 990
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  18. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  19. 2019Solving Rubik's Cube with a Robot Handcited by 699
  20. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  21. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  22. 2020Jukebox: A Generative Model for Musiccited by 535
  23. 2019Model-Based Reinforcement Learning for Ataricited by 495
  24. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  25. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  26. 2022MAGVIT: Masked Generative Video Transformercited by 341
  27. 2022Transformers are Sample-Efficient World Modelscited by 317
  28. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  29. 2016Using the Output Embedding to Improve Language Modelscited by 266
  30. 2021Urban Radiance Fieldscited by 263
  31. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  32. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  33. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  34. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  35. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  36. 2023NerfAcc: Efficient Sampling Accelerates NeRFscited by 90
  37. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  38. 2022Improved Masked Image Generation with Token-Criticcited by 63

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 32; 2025: 45; 2026: 21.

105 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  2. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  3. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  5. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  6. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  7. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  8. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  9. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  10. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Drivingcited by 89
  11. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  12. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
38 more
  1. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  2. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  4. 20253D and 4D World Modeling: A Surveycited by 64
  5. 2025A Survey of World Models for Autonomous Drivingcited by 60
  6. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  7. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  8. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  9. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  10. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  11. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  12. 2024Policy-Guided Diffusioncited by 40
  13. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  14. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  15. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  16. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  17. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  18. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  19. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  20. 2025Halton Scheduler For Masked Generative Image Transformercited by 30
  21. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  22. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Drivingcited by 29
  23. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  24. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  25. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  26. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  27. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  28. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  29. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  30. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  31. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  32. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  33. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  34. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  35. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  36. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  37. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  38. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning
  2. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
  3. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
  4. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
  5. 2026UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
  6. 2026Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusion
  7. 2026WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving
  8. 2024GPD-1: Generative Pre-training for Driving