mlsanityData to 14 Oct 2026

Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents

arXiv 2402.05746 ↗First posted 8 Feb 2024cs.CVcited by 91

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

Scene simulation in autonomous driving has gained significant attention because of its huge potential for generating customized data. However, existing editable scene simulation approaches face limitations in terms of user interaction efficiency, multi-camera photo-realistic rendering and external digital assets integration. To address these challenges, this paper introduces ChatSim, the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. To enable editing with high command flexibility,~ChatSim leverages a large language model (LLM) agent collaboration framework. To generate photo-realistic outcomes, ChatSim employs a novel multi-camera neural radiance field method. Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering. Our experiments on Waymo Open Dataset demonstrate that ChatSim can handle complex language commands and generate corresponding photo-realistic scene videos.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 61 of its 82 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

61 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Segment Anythingcited by 8,234
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
38 more
  1. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  7. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  8. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  9. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  10. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  12. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 314
  13. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  14. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  15. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  16. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  17. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 231
  18. 2023Playing repeated games with Large Language Modelscited by 217
  19. 2023MapTRv2: An End-to-End Framework for Online Vectorized HD Map Constructioncited by 193
  20. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  21. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  22. 2020RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Drivingcited by 150
  23. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 121
  24. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  25. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  26. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  27. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  28. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  29. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  30. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  31. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  32. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  33. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 73
  34. 2020SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Drivingcited by 68
  35. 2019Deep Sky Modeling for Single Image Outdoor Lighting Estimationcited by 66
  36. 2019DeepLight: Learning Illumination for Unconstrained Mobile Mixed Realitycited by 63
  37. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  38. 2016SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Drivingcited by 55

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 31; 2025: 42; 2026: 18.

91 papers.

Sort:

By area:

  1. 2024OmniRe: Omni Urban Scene Reconstructioncited by 161
  2. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 122
  3. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 116
  4. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 84
  5. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  6. 2024SA-GS: Scale-Adaptive Gaussian Splatting for Training-Free Anti-Aliasingcited by 41
  7. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  8. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  9. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  10. 2024Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenescited by 30
  11. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  12. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMscited by 24
38 more
  1. 2024Rip-NeRF: Anti-aliasing Radiance Fields with Ripmap-Encoded Platonic Solidscited by 23
  2. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  3. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  4. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  5. 2025PharmAgents: Building a Virtual Pharma with Large Language Model Agentscited by 21
  6. 2024Drone-assisted Road Gaussian Splatting with Cross-view Uncertaintycited by 20
  7. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  8. 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehiclescited by 18
  9. 2024DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenescited by 17
  10. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  11. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 16
  12. 2024Training-Free Model Merging for Multi-target Domain Adaptationcited by 15
  13. 2024LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splattingcited by 15
  14. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 15
  15. 2024An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Makingcited by 14
  16. 2024Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputscited by 12
  17. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  18. 2025SceneCrafter: Controllable Multi-View Driving Scene Editingcited by 9
  19. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 9
  20. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  21. 2024MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaborationcited by 8
  22. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 8
  23. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  24. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  25. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 7
  26. 2025Persistent Autoregressive Mapping with Traffic Rules for Autonomous Drivingcited by 6
  27. 2026Engine-Native Editable 3D World Reconstruction with Objects and Lightingcited by 6
  28. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  29. 2025NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perceptioncited by 5
  30. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 4
  31. 2024ViUniT: Visual Unit Tests for More Robust Visual Programmingcited by 4
  32. 2025MObI: Multimodal Object Inpainting Using Diffusion Modelscited by 4
  33. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  34. 2025PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animationcited by 4
  35. 2025DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenescited by 4
  36. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  37. 2024Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Reviewcited by 3
  38. 2024Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modelingcited by 3

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025When Digital Twins Meet Large Language Models: Realistic, Interactive, and Editable Simulation for Autonomous Driving
  2. 2026Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes
  3. 2024Probing Multimodal LLMs as World Models for Driving
  4. 2025SIMSplat: Language-Aligned 4D Gaussian Splatting for Driving Scenario Generation
  5. 2025RealEngine: Simulating Autonomous Driving in Realistic Context
  6. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model
  7. 2026LychSim: A Controllable and Interactive Simulation Framework for Vision Research
  8. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving