# Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models

arXiv: https://arxiv.org/abs/2406.14712 · HTML: https://mlsanity.com/paper/2406.14712/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [Granite Code Models: A Family of Open Foundation Models for Code Intelligence](https://mlsanity.com/paper/2405.04324/) (2024)
- [Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions](https://mlsanity.com/paper/2312.12450/) (2023)
- [Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code](https://mlsanity.com/paper/2405.19495/) (2024)

## Built on by (most cited first, 30 of 30)

- [Agent-Q: Fine-Tuning Large Language Models for Quantum Circuit Generation and Optimization](https://mlsanity.com/paper/2504.11109/) (2025)
- [Enhancing LLM-based Quantum Code Generation with Multi-Agent Optimization and Quantum Error Correction](https://mlsanity.com/paper/2504.14557/) (2025)
- [QAgent: An LLM-based Multi-Agent System for Autonomous OpenQASM programming](https://mlsanity.com/paper/2508.20134/) (2025)
- [QUASAR: Quantum Assembly Code Generation Using Tool-Augmented LLMs via Agentic RL](https://mlsanity.com/paper/2510.00967/) (2025)
- [QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges](https://mlsanity.com/paper/2506.20008/) (2025)
- [QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback](https://mlsanity.com/paper/2510.26101/) (2025)
- [PennyCoder: Efficient Domain-Specific LLMs for PennyLane-Based Quantum Code Generation](https://mlsanity.com/paper/2507.19562/) (2025)
- [Quantum Verifiable Rewards for Post-Training Qiskit Code Assistant](https://mlsanity.com/paper/2508.20907/) (2025)
- [Thompson Sampling via Fine-Tuning of LLMs](https://mlsanity.com/paper/2510.13328/) (2025)
- [QuantumBench: A Benchmark for Quantum Problem Solving](https://mlsanity.com/paper/2511.00092/) (2025)
- [A Theoretical Analysis of Test-Driven Code Generation](https://mlsanity.com/paper/2602.06098/) (2026)
- [QSpark: Towards Reliable Qiskit Code Generation](https://mlsanity.com/paper/2507.12642/) (2025)
- [QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation](https://mlsanity.com/paper/2604.08570/) (2026)
- [Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation](https://mlsanity.com/paper/2605.27210/) (2026)
- [QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging](https://mlsanity.com/paper/2606.07314/) (2026)
- [AI+HW 2035: Shaping the Next Decade](https://mlsanity.com/paper/2603.05225/) (2026)
- [PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation](https://mlsanity.com/paper/2605.25572/) (2026)
- [Aligning Quantum Operators with Large Language Models](https://mlsanity.com/paper/2606.13811/) (2026)
- [Architecture for a Trustworthy Quantum Chatbot](https://mlsanity.com/paper/2503.04875/) (2025)
- [Generative AI for Quantum Circuits and Quantum Code: A Technical Review and Taxonomy](https://mlsanity.com/paper/2603.16216/) (2026)
- [POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles](https://mlsanity.com/paper/2605.07775/) (2026)
- [QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits](https://mlsanity.com/paper/2605.30358/) (2026)
- [Leveraging LLM-Based Agentic Systems to Generate Quantum Applications for Test Optimization](https://mlsanity.com/paper/2607.00939/) (2026)
- [Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions](https://mlsanity.com/paper/2607.04072/) (2026)
- [Position: Quantum Program Generation Must Prioritize Validity Over Probabilistic Scaling](https://mlsanity.com/paper/2607.15313/) (2026)
- [Hybrid LLM-Guided Search for Quantum Reservoir Architecture Design](https://mlsanity.com/paper/2607.19506/) (2026)
- [LLM-Driven Algorithm Design for Quantum Circuit Synthesis based on Binary Decision Diagrams](https://mlsanity.com/paper/2609.05327/) (2026)
- [QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?](https://mlsanity.com/paper/2609.20319/) (2026)
- [QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks](https://mlsanity.com/paper/2609.35581/) (2026)
- [QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code](https://mlsanity.com/paper/2610.13811/) (2026)

## Abstract (arXiv)

Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.