# Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

arXiv: https://arxiv.org/abs/2302.08418 · HTML: https://mlsanity.com/paper/2302.08418/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [DriveGAN: Towards a Controllable High-Quality Neural Simulation](https://mlsanity.com/paper/2104.15060/) (2021)
- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving](https://mlsanity.com/paper/2005.03844/) (2020)
- [Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities](https://mlsanity.com/paper/2202.02559/) (2022)
- [A Full Dive into Realizing the Edge-enabled Metaverse: Visions, Enabling Technologies,and Challenges](https://mlsanity.com/paper/2203.05471/) (2022)
- [Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directions](https://mlsanity.com/paper/2102.12169/) (2021)
- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [Massive Access for 5G and Beyond](https://mlsanity.com/paper/2002.03491/) (2020)
- [Saliency in Augmented Reality](https://mlsanity.com/paper/2204.08308/) (2022)
- [Reliable Distributed Computing for Metaverse: A Hierarchical Game-Theoretic Approach](https://mlsanity.com/paper/2111.10548/) (2021)
- [Generative Neural Network Channel Modeling for Millimeter-Wave UAV Communication](https://mlsanity.com/paper/2012.09133/) (2020)
- [Experience: Understanding Long-Term Evolving Patterns of Shared Electric Vehicle Networks](https://mlsanity.com/paper/1812.07499/) (2018)
- [Multi-Dimensional Auction Mechanisms for Crowdsourced Mobile Video Streaming](https://mlsanity.com/paper/1703.06648/) (2017)
- [Non-Cash Auction for Spectrum Trading in Cognitive Radio Networks: A Contract Theoretical Model with Joint Adverse Selection and Moral Hazard](https://mlsanity.com/paper/1709.08317/) (2017)
- [EPViSA: Efficient Auction Design for Real-time Physical-Virtual Synchronization in the Metaverse](https://mlsanity.com/paper/2211.06838/) (2022)
- [Digital Twin-Driven Computing Resource Management for Vehicular Networks](https://mlsanity.com/paper/2211.13818/) (2022)
- [QoE Driven VR 360 Video Massive MIMO Transmission](https://mlsanity.com/paper/2106.08165/) (2021)

## Built on by (most cited first, 22 of 22)

- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [Generative AI on SpectrumNet: An Open Benchmark of Multiband 3D Radio Maps](https://mlsanity.com/paper/2408.15252/) (2024)
- [VesselVAE: Recursive Variational Autoencoders for 3D Blood Vessel Synthesis](https://mlsanity.com/paper/2307.03592/) (2023)
- [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024)
- [An Overview on Generative AI at Scale with Edge-Cloud Computing](https://mlsanity.com/paper/2306.17170/) (2023)
- [Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/2405.20024/) (2024)
- [End-to-end Autonomous Driving using Deep Learning: A Systematic Review](https://mlsanity.com/paper/2311.18636/) (2023)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [Multi-attribute Auction-based Resource Allocation for Twins Migration in Vehicular Metaverses: A GPT-based DRL Approach](https://mlsanity.com/paper/2406.05418/) (2024)
- [A Gentle Introduction and Tutorial on Deep Generative Models in Transportation Research](https://mlsanity.com/paper/2410.07066/) (2024)
- [Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey](https://mlsanity.com/paper/2310.10665/) (2023)
- [Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligence](https://mlsanity.com/paper/2312.16497/) (2023)
- [Multi-Agent RL-Based Industrial AIGC Service Offloading over Wireless Edge Networks](https://mlsanity.com/paper/2405.02972/) (2024)
- [Scalable AI Generative Content for Vehicular Network Semantic Communication](https://mlsanity.com/paper/2311.13782/) (2023)
- [Large AI Model for Delay-Doppler Domain Channel Prediction in 6G OTFS-Based Vehicular Networks](https://mlsanity.com/paper/2503.01116/) (2025)
- [On the Performance of an Explainable Language Model on PubMedQA](https://mlsanity.com/paper/2504.05074/) (2025)
- [Confidence-Regulated Generative Diffusion Models for Reliable AI Agent Migration in Vehicular Metaverses](https://mlsanity.com/paper/2505.12710/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [High Efficiency Inference Accelerating Algorithm for NOMA-based Mobile Edge Computing](https://mlsanity.com/paper/2312.15850/) (2023)
- [A QoE-Aware Split Inference Accelerating Algorithm for NOMA-based Edge Intelligence](https://mlsanity.com/paper/2409.16537/) (2024)
- [ILASH: A Predictive Neural Architecture Search Framework for Multi-Task Applications](https://mlsanity.com/paper/2412.02116/) (2024)
- [Recursive Variational Autoencoders for 3D Blood Vessel Generative Modeling](https://mlsanity.com/paper/2506.14914/) (2025)

## Abstract (arXiv)

In the vehicular mixed reality (MR) Metaverse, the distance between physical and virtual entities can be overcome by fusing the physical and virtual environments with multi-dimensional communications in autonomous driving systems. Assisted by digital twin (DT) technologies, connected autonomous vehicles (AVs), roadside units (RSU), and virtual simulators can maintain the vehicular MR Metaverse via digital simulations for sharing data and making driving decisions collaboratively. However, large-scale traffic and driving simulation via realistic data collection and fusion from the physical world for online prediction and offline training in autonomous driving systems are difficult and costly. In this paper, we propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations for improving driving safety and traffic efficiency. First, we propose a multi-task DT offloading model for the reliable execution of heterogeneous DT tasks with different requirements at RSUs. Then, based on the preferences of AV's DTs and collected realistic data, virtual simulators can synthesize unlimited conditioned driving and traffic datasets to further improve robustness. Finally, we propose a multi-task enhanced auction-based mechanism to provide fine-grained incentives for RSUs in providing resources for autonomous driving. The property analysis and experimental results demonstrate that the proposed mechanism and architecture are strategy-proof and effective, respectively.