# DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant

arXiv: https://arxiv.org/abs/2604.12615 · HTML: https://mlsanity.com/paper/2604.12615/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 12 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 3 of 3)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [STELLAR: A Search-Based Testing Framework for Large Language Model Applications](https://mlsanity.com/paper/2601.00497/) (2026)

## Built on by (most cited first, 1 of 1)

- [From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization](https://mlsanity.com/paper/2609.19630/) (2026)

## Abstract (arXiv)

This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testing solutions were evaluated based on their effectiveness in exposing failures and the diversity of the discovered failure-revealing tests. We report on the experimental methodology, the competitors, and the results.