# QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

arXiv: https://arxiv.org/abs/2609.35581 · HTML: https://mlsanity.com/paper/2609.35581/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models](https://mlsanity.com/paper/2406.14712/) (2024)
- [Agents for self-driving laboratories applied to quantum computing](https://mlsanity.com/paper/2412.07978/) (2024)
- [QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design](https://mlsanity.com/paper/2410.07961/) (2024)
- [QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback](https://mlsanity.com/paper/2510.26101/) (2025)
- [Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing](https://mlsanity.com/paper/2602.10092/) (2026)
- [CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers](https://mlsanity.com/paper/2510.05228/) (2025)
- [Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis](https://mlsanity.com/paper/2602.24060/) (2026)
- [Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation](https://mlsanity.com/paper/2605.27210/) (2026)

## Abstract (arXiv)

We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models × 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.