# CarDreamer: Open-Source Learning Platform for World Model based Autonomous Driving

arXiv: https://arxiv.org/abs/2405.09111 · HTML: https://mlsanity.com/paper/2405.09111/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 15 May 2024.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)
- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)

## Builds on (most cited first, 30 of 31)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023)
- [DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation](https://mlsanity.com/paper/2403.06845/) (2024)
- [Model-Based Imitation Learning for Urban Driving](https://mlsanity.com/paper/2210.07729/) (2022)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [ADriver-I: A General World Model for Autonomous Driving](https://mlsanity.com/paper/2311.13549/) (2023)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer](https://mlsanity.com/paper/2304.10628/) (2023)
- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models](https://mlsanity.com/paper/2205.13817/) (2022)
- [Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)](https://mlsanity.com/paper/2402.16720/) (2024)
- [Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model](https://mlsanity.com/paper/2210.04017/) (2022)
- [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones](https://mlsanity.com/paper/2211.09703/) (2022)

## Built on by (most cited first, 13 of 13)

- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training](https://mlsanity.com/paper/2503.12030/) (2025)
- [DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving](https://mlsanity.com/paper/2602.06521/) (2026)
- [GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control](https://mlsanity.com/paper/2505.22421/) (2025)
- [VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving](https://mlsanity.com/paper/2505.16377/) (2025)
- [World Model Robustness via Surprise Recognition](https://mlsanity.com/paper/2512.01119/) (2025)
- [TrafficGamer: Reliable and Flexible Traffic Simulation for Safety-Critical Scenarios with Game-Theoretic Oracles](https://mlsanity.com/paper/2408.15538/) (2024)
- [EI-Drive: A Platform for Cooperative Perception with Realistic Communication Models](https://mlsanity.com/paper/2412.09782/) (2024)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception](https://mlsanity.com/paper/2604.17651/) (2026)
- [Vehicle Dynamics Embedded World Models for Autonomous Driving](https://mlsanity.com/paper/2512.02417/) (2025)
- [Sim-to-Real Aware End-to-End Learning Environment for Micromobility](https://mlsanity.com/paper/2609.28969/) (2026)

## Abstract (arXiv)

To safely navigate intricate real-world scenarios, autonomous vehicles must be able to adapt to diverse road conditions and anticipate future events. World model (WM) based reinforcement learning (RL) has emerged as a promising approach by learning and predicting the complex dynamics of various environments. Nevertheless, to the best of our knowledge, there does not exist an accessible platform for training and testing such algorithms in sophisticated driving environments. To fill this void, we introduce CarDreamer, the first open-source learning platform designed specifically for developing WM based autonomous driving algorithms. It comprises three key components: 1) World model backbone: CarDreamer has integrated some state-of-the-art WMs, which simplifies the reproduction of RL algorithms. The backbone is decoupled from the rest and communicates using the standard Gym interface, so that users can easily integrate and test their own algorithms. 2) Built-in tasks: CarDreamer offers a comprehensive set of highly configurable driving tasks which are compatible with Gym interfaces and are equipped with empirically optimized reward functions. 3) Task development suite: This suite streamlines the creation of driving tasks, enabling easy definition of traffic flows and vehicle routes, along with automatic collection of multi-modal observation data. A visualization server allows users to trace real-time agent driving videos and performance metrics through a browser. Furthermore, we conduct extensive experiments using built-in tasks to evaluate the performance and potential of WMs in autonomous driving. Thanks to the richness and flexibility of CarDreamer, we also systematically study the impact of observation modality, observability, and sharing of vehicle intentions on AV safety and efficiency. All code and documents are accessible on https://github.com/ucd-dare/CarDreamer.