mlsanityData to 14 Oct 2026

RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios

arXiv 2312.13303 ↗First posted 19 Dec 2023cs.LG cs.AIcited by 46

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Simulation plays a crucial role in the development of autonomous vehicles (AVs) due to the potential risks associated with real-world testing. Although significant progress has been made in the visual aspects of simulators, generating complex behavior among agents remains a formidable challenge. It is not only imperative to ensure realism in the scenarios generated but also essential to incorporate preferences and conditions to facilitate controllable generation for AV training and evaluation. Traditional methods, mainly relying on memorizing the distribution of training datasets, often fall short in generating unseen scenarios. Inspired by the success of retrieval augmented generation in large language models, we present RealGen, a novel retrieval-based in-context learning framework for traffic scenario generation. RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way, which may originate from templates or tagged scenarios. This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios, compose various behaviors, and produce critical scenarios. Evaluations show that RealGen offers considerable flexibility and controllability, marking a new direction in the field of controllable traffic scenario generation. Check our project website for more information: https://realgen.github.io.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 50 of its 67 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

50 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generator29 Sep

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2016Layer Normalizationcited by 5,669
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
37 more
  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  10. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 270
  11. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 193
  12. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 190
  13. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 171
  14. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 160
  15. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  16. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
  17. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  18. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  19. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 106
  20. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 96
  21. 2023Language Conditioned Traffic Generationcited by 87
  22. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 84
  23. 2021Self-supervised learning methods and applications in medical imaging analysis: A surveycited by 77
  24. 2020Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Methodcited by 75
  25. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  26. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 73
  27. 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
  28. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  29. 2020Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluationcited by 55
  30. 2022AdvDO: Realistic Adversarial Attacks for Trajectory Predictioncited by 49
  31. 2022Retrieval-based Controllable Molecule Generationcited by 44
  32. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  33. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 42
  34. 2023trajdata: A Unified Interface to Multiple Human Trajectory Datasetscited by 25
  35. 2022Relational Memory Augmented Language Modelscited by 24
  36. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  37. 2023RMP: A Random Mask Pretrain Framework for Motion Predictioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 18; 2026: 13.

46 papers.

Sort:

By area:

  1. 2025A Survey of World Models for Autonomous Drivingcited by 60
  2. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  3. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  4. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 27
  5. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  6. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 21
  7. 2024Probing Multimodal LLMs as World Models for Drivingcited by 20
  8. 2024Promptable Closed-loop Traffic Simulationcited by 20
  9. 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehiclescited by 18
  10. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  11. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  12. 2024Learning to Drive via Asymmetric Self-Playcited by 11
34 more
  1. 2024Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Modelcited by 11
  2. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  3. 2026World Engine: Towards the Era of Post-Training for Autonomous Drivingcited by 8
  4. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  5. 2025Continuous Locomotive Crowd Behavior Generationcited by 7
  6. 2025Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Modelscited by 7
  7. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  8. 2024SynthAI: A Multi Agent Generative AI Framework for Automated Modular HLS Design Generationcited by 6
  9. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectivescited by 5
  10. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Frameworkcited by 5
  11. 2025RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulationcited by 4
  12. 2024Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Reviewcited by 3
  13. 2024ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenescited by 3
  14. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  15. 2025RealDrive: Retrieval-Augmented Driving with Diffusion Modelscited by 3
  16. 2025SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integrationcited by 3
  17. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 3
  18. 2026CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generationcited by 3
  19. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  20. 2025Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generationcited by 2
  21. 2025SceneStreamer: Continuous Scenario Generation as Next Token Group Predictioncited by 2
  22. 2026Virtual Traffic Police: Large Language Model-Augmented Traffic Signal Control for Unforeseen Incidentscited by 1
  23. 2025AI-Driven Climate Policy Scenario Generation for Sub-Saharan Africacited by 0
  24. 2025UrbanMind: Towards Urban General Intelligence via Tool-Enhanced Retrieval-Augmented Generation and Multilevel Optimizationcited by 0
  25. 2025SocialDriveGen: Generating Diverse Traffic Scenarios with Controllable Social Interactionscited by 0
  26. 2026Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehiclescited by 0
  27. 2026Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Trafficcited by 0
  28. 2026Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Trainingcited by 0
  29. 2026Revealing Safety-Critical Scenarios for UTM via Transformercited by 0
  30. 2026ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulationcited by 0
  31. 2026End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generationcited by 0
  32. 2026Artificial Intelligence and Modeling & Simulation: An Overviewcited by 0
  33. 2026TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrievalcited by 0
  34. 2026BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion
  2. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior
  3. 2026Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving
  4. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles
  5. 2024Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation
  6. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
  7. 2023(Re)2H2O: Autonomous Driving Scenario Generation via Reversely Regularized Hybrid Offline-and-Online Reinforcement Learning
  8. 2025Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge