# Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

arXiv: https://arxiv.org/abs/2307.11769 · HTML: https://mlsanity.com/paper/2307.11769/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Ontology based Scene Creation for the Development of Automated Vehicles](https://mlsanity.com/paper/1704.01006/) (2017)
- [A Survey on Automated Driving System Testing: Landscapes and Trends](https://mlsanity.com/paper/2206.05961/) (2022)
- [A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation](https://mlsanity.com/paper/2112.00964/) (2021)
- [BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems](https://mlsanity.com/paper/2307.07493/) (2023)
- [Automatic Map Generation for Autonomous Driving System Testing](https://mlsanity.com/paper/2206.09357/) (2022)
- [FLYOVER: A Model-Driven Method to Generate Diverse Highway Interchanges for Autonomous Vehicle Testing](https://mlsanity.com/paper/2301.12738/) (2023)

## Built on by (most cited first, 8 of 8)

- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent](https://mlsanity.com/paper/2311.18307/) (2023)
- [Large-scale moral machine experiment on large language models](https://mlsanity.com/paper/2411.06790/) (2024)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Towards a Common Framework for Autoformalization](https://mlsanity.com/paper/2509.09810/) (2025)
- [TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning](https://mlsanity.com/paper/2403.08694/) (2024)
- [MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery](https://mlsanity.com/paper/2402.18169/) (2024)

## Abstract (arXiv)

Engineering knowledge-based (or expert) systems require extensive manual effort and domain knowledge. As Large Language Models (LLMs) are trained using an enormous amount of cross-domain knowledge, it becomes possible to automate such engineering processes. This paper presents an empirical automation and semi-automation framework for domain knowledge distillation using prompt engineering and the LLM ChatGPT. We assess the framework empirically in the autonomous driving domain and present our key observations. In our implementation, we construct the domain knowledge ontology by "chatting" with ChatGPT. The key finding is that while fully automated domain ontology construction is possible, human supervision and early intervention typically improve efficiency and output quality as they lessen the effects of response randomness and the butterfly effect. We, therefore, also develop a web-based distillation assistant enabling supervision and flexible intervention at runtime. We hope our findings and tools could inspire future research toward revolutionizing the engineering of knowledge-based systems across application domains.