# Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehicles

arXiv: https://arxiv.org/abs/2410.09829 · HTML: https://mlsanity.com/paper/2410.09829/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [FRANC: A Lightweight Framework for High-Quality Code Generation](https://mlsanity.com/paper/2307.08220/) (2023)
- [Exploring Data Augmentation for Code Generation Tasks](https://mlsanity.com/paper/2302.03499/) (2023)
- [Dialogue-based generation of self-driving simulation scenarios using Large Language Models](https://mlsanity.com/paper/2310.17372/) (2023)
- [Memory-Based Semantic Parsing](https://mlsanity.com/paper/2110.07358/) (2021)

## Built on by (most cited first, 4 of 4)

- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [CrashAgent: Crash Scenario Generation via Multi-modal Reasoning](https://mlsanity.com/paper/2505.18341/) (2025)
- [David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation](https://mlsanity.com/paper/2510.14115/) (2025)
- [Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving](https://mlsanity.com/paper/2607.14387/) (2026)

## Abstract (arXiv)

Cyber-physical systems like autonomous vehicles are tested in simulation before deployment, using domain-specific programs for scenario specification. To aid the testing of autonomous vehicles in simulation, we design a natural language interface, using an instruction-following large language model, to assist a non-coding domain expert in synthesising the desired scenarios and vehicle behaviours. We show that using it to convert utterances to the symbolic program is feasible, despite the very small training dataset. Human experiments show that dialogue is critical to successful simulation generation, leading to a 4.5 times higher success rate than a generation without engaging in extended conversation.