# QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

arXiv: https://arxiv.org/abs/2604.25884 · HTML: https://mlsanity.com/paper/2604.25884/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild](https://mlsanity.com/paper/2407.04172/) (2024)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2403.09028/) (2024)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting](https://mlsanity.com/paper/2303.14100/) (2023)
- [VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning](https://mlsanity.com/paper/2403.13164/) (2024)
- [Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs](https://mlsanity.com/paper/2406.00257/) (2024)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [Towards Multimodal In-Context Learning for Vision & Language Models](https://mlsanity.com/paper/2403.12736/) (2024)
- [Agents for self-driving laboratories applied to quantum computing](https://mlsanity.com/paper/2412.07978/) (2024)

## Built on by (most cited first, 11 of 11)

- [Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation](https://mlsanity.com/paper/2605.27210/) (2026)
- [Agentic AI for Scientific Reasoning in Autonomous Quantum Sensing Experiments](https://mlsanity.com/paper/2607.25145/) (2026)
- [RIZZ: Routing Interactions to Near Zero-Interference Zones for Continual Adaptation of Black-Box Agents](https://mlsanity.com/paper/2606.20638/) (2026)
- [Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI](https://mlsanity.com/paper/2610.09146/) (2026)
- [RiverONE: Generating Knowledge-Intensive VLM by Simulated Quantum Machines](https://mlsanity.com/paper/2606.29966/) (2026)
- [When AI meets quantum information: A comprehensive review](https://mlsanity.com/paper/2607.00365/) (2026)
- [Self-Specializing Vision-Language Transmon Chip Calibration in a Physics-Grounded Environment](https://mlsanity.com/paper/2607.03193/) (2026)
- [Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning](https://mlsanity.com/paper/2607.09422/) (2026)
- [Quantum Circuit Vision: Cost-Aware Evaluation of Visual AI Agents for Quantum Code Generation](https://mlsanity.com/paper/2607.10057/) (2026)
- [Machine Learning for Charge State Characterization of Isolated Double Quantum Dots](https://mlsanity.com/paper/2607.20871/) (2026)
- [Evaluating Verified Autonomy in Quantum Engineering](https://mlsanity.com/paper/2609.17439/) (2026)

## Abstract (arXiv)

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.