QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 12 of its 30 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
12 of 30 references.
References · 12
- 2023GPT-4 Technical Reportcited by 18,396
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2024Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Modelscited by 30
- 2024Qiskit Code Assistant: Training LLMs for generating Quantum Computing Codecited by 25
- 2024QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Designcited by 15
- 2025QuanBench: Benchmarking Quantum Code Generation with Large Language Modelscited by 12
- 2025QUASAR: Quantum Assembly Code Generation Using Tool-Augmented LLMs via Agentic RLcited by 11
- 2025QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challengescited by 8
- 2025QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedbackcited by 7
- 2025PennyCoder: Efficient Domain-Specific LLMs for PennyLane-Based Quantum Code Generationcited by 6
- 2025Quantum Verifiable Rewards for Post-Training Qiskit Code Assistantcited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.