# QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

arXiv: https://arxiv.org/abs/2610.13811 · HTML: https://mlsanity.com/paper/2610.13811/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 17 of 17)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [CodeScore: Evaluating Code Generation by Learning Code Execution](https://mlsanity.com/paper/2301.09043/) (2023)
- [Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models](https://mlsanity.com/paper/2406.14712/) (2024)
- [Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code](https://mlsanity.com/paper/2405.19495/) (2024)
- [Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review](https://mlsanity.com/paper/2406.12655/) (2024)
- [QuanBench: Benchmarking Quantum Code Generation with Large Language Models](https://mlsanity.com/paper/2510.16779/) (2025)
- [Enhancing LLM-based Quantum Code Generation with Multi-Agent Optimization and Quantum Error Correction](https://mlsanity.com/paper/2504.14557/) (2025)
- [A PennyLane-Centric Dataset to Enhance LLM-based Quantum Code Generation using RAG](https://mlsanity.com/paper/2503.02497/) (2025)
- [QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges](https://mlsanity.com/paper/2506.20008/) (2025)
- [Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis](https://mlsanity.com/paper/2508.14727/) (2025)
- [QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback](https://mlsanity.com/paper/2510.26101/) (2025)
- [On Testing Quantum Programs](https://mlsanity.com/paper/1812.09261/) (2018)
- [PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation](https://mlsanity.com/paper/2605.25572/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.