# Automating a Complete Software Test Process Using LLMs: An Automotive Case Study

arXiv: https://arxiv.org/abs/2502.04008 · HTML: https://mlsanity.com/paper/2502.04008/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 15 of 15)

- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023)
- [Leveraging Large Language Models to Improve REST API Testing](https://mlsanity.com/paper/2312.00894/) (2023)
- [Towards Autonomous Testing Agents via Conversational Large Language Models](https://mlsanity.com/paper/2306.05152/) (2023)
- [Testing RESTful APIs: A Survey](https://mlsanity.com/paper/2212.14604/) (2022)
- [APITestGenie: Automated API Test Generation through Generative AI](https://mlsanity.com/paper/2409.03838/) (2024)
- [KAT: Dependency-aware Automated API Testing with Large Language Models](https://mlsanity.com/paper/2407.10227/) (2024)
- [Pythia: Grammar-Based Fuzzing of REST APIs with Coverage-guided Feedback and Learning-based Mutations](https://mlsanity.com/paper/2005.11498/) (2020)
- [Automating REST API Postman Test Cases Using LLM](https://mlsanity.com/paper/2404.10678/) (2024)
- [Improving Test Case Generation for REST APIs Through Hierarchical Clustering](https://mlsanity.com/paper/2109.06655/) (2021)
- [foREST: A Tree-based Approach for Fuzzing RESTful APIs](https://mlsanity.com/paper/2203.02906/) (2022)

## Built on by (most cited first, 8 of 8)

- [Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy](https://mlsanity.com/paper/2503.00481/) (2025)
- [The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE](https://mlsanity.com/paper/2604.15468/) (2026)
- [Workflow-Level Design Principles for Trustworthy GenAI in Automotive System Engineering](https://mlsanity.com/paper/2602.19614/) (2026)
- [Still Manual? Automated Linter Configuration via DSL-Based LLM Compilation of Coding Standards](https://mlsanity.com/paper/2602.07783/) (2026)
- [LLM-Powered Workflow Optimization for Multidisciplinary Software Development: An Automotive Industry Case Study](https://mlsanity.com/paper/2603.21439/) (2026)
- [Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation](https://mlsanity.com/paper/2604.15270/) (2026)
- [Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements](https://mlsanity.com/paper/2606.17197/) (2026)
- [ARIA - An Agentic Framework for Autonomous Testing of Infotainment Systems](https://mlsanity.com/paper/2609.04913/) (2026)

## Abstract (arXiv)

Vehicle API testing verifies whether the interactions between a vehicle's internal systems and external applications meet expectations, ensuring that users can access and control various vehicle functions and data. However, this task is inherently complex, requiring the alignment and coordination of API systems, communication protocols, and even vehicle simulation systems to develop valid test cases. In practical industrial scenarios, inconsistencies, ambiguities, and interdependencies across various documents and system specifications pose significant challenges. This paper presents a system designed for the automated testing of in-vehicle APIs. By clearly defining and segmenting the testing process, we enable Large Language Models (LLMs) to focus on specific tasks, ensuring a stable and controlled testing workflow. Experiments conducted on over 100 APIs demonstrate that our system effectively automates vehicle API testing. The results also confirm that LLMs can efficiently handle mundane tasks requiring human judgment, making them suitable for complete automation in similar industrial contexts.