mlsanityData to 7 Oct 2026

Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models

arXiv 2406.14712 ↗First posted 20 Jun 2024quant-ph cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

12 of 25 references.

References · 12

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  5. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  8. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  9. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  10. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  11. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 46
  12. 2024Qiskit Code Assistant: Training LLMs for generating Quantum Computing Codecited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 17.

29 papers.

Sort:

By area:

  1. 2025Agent-Q: Fine-Tuning Large Language Models for Quantum Circuit Generation and Optimizationcited by 17
  2. 2025QUASAR: Quantum Assembly Code Generation Using Tool-Augmented LLMs via Agentic RLcited by 11
  3. 2025Enhancing LLM-based Quantum Code Generation with Multi-Agent Optimization and Quantum Error Correctioncited by 10
  4. 2025QAgent: An LLM-based Multi-Agent System for Autonomous OpenQASM programmingcited by 9
  5. 2025QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challengescited by 7
  6. 2025PennyCoder: Efficient Domain-Specific LLMs for PennyLane-Based Quantum Code Generationcited by 6
  7. 2025Quantum Verifiable Rewards for Post-Training Qiskit Code Assistantcited by 6
  8. 2025Thompson Sampling via Fine-Tuning of LLMscited by 6
  9. 2025QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedbackcited by 6
  10. 2025QuantumBench: A Benchmark for Quantum Problem Solvingcited by 3
  11. 2026A Theoretical Analysis of Test-Driven Code Generationcited by 2
  12. 2026QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generationcited by 2
17 more
  1. 2026Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluationcited by 2
  2. 2026QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debuggingcited by 2
  3. 2025QSpark: Towards Reliable Qiskit Code Generationcited by 1
  4. 2026AI+HW 2035: Shaping the Next Decadecited by 1
  5. 2026Aligning Quantum Operators with Large Language Modelscited by 1
  6. 2025Architecture for a Trustworthy Quantum Chatbotcited by 0
  7. 2026Generative AI for Quantum Circuits and Quantum Code: A Technical Review and Taxonomycited by 0
  8. 2026POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensemblescited by 0
  9. 2026PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generationcited by 0
  10. 2026QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuitscited by 0
  11. 2026Leveraging LLM-Based Agentic Systems to Generate Quantum Applications for Test Optimizationcited by 0
  12. 2026Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versionscited by 0
  13. 2026Position: Quantum Program Generation Must Prioritize Validity Over Probabilistic Scalingcited by 0
  14. 2026Hybrid LLM-Guided Search for Quantum Reservoir Architecture Designcited by 0
  15. 2026LLM-Driven Algorithm Design for Quantum Circuit Synthesis based on Binary Decision Diagramscited by 0
  16. 2026QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?cited by 0
  17. 2026QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Taskscited by 0