# Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark

arXiv: https://arxiv.org/abs/2504.16137 · HTML: https://mlsanity.com/paper/2504.16137/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Builds on (most cited first, 30 of 33)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents](https://mlsanity.com/paper/2410.13886/) (2024)
- [CRISPR-GPT for Agentic Automation of Gene-editing Experiments](https://mlsanity.com/paper/2404.18021/) (2024)
- [Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools](https://mlsanity.com/paper/2306.13952/) (2023)
- [How predictable is language model benchmark performance?](https://mlsanity.com/paper/2401.04757/) (2024)
- [Will releasing the weights of future large language models grant widespread access to pandemic agents?](https://mlsanity.com/paper/2310.18233/) (2023)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs](https://mlsanity.com/paper/2409.02257/) (2024)
- [Introduction to AI Safety, Ethics, and Society](https://mlsanity.com/paper/2411.01042/) (2024)
- [Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers](https://mlsanity.com/paper/2410.11672/) (2024)

## Built on by (most cited first, 30 of 31)

- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [FORTRESS: Frontier Risk Evaluation for National Security and Public Safety](https://mlsanity.com/paper/2506.14922/) (2025)
- [PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach](https://mlsanity.com/paper/2511.20703/) (2025)
- [LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring](https://mlsanity.com/paper/2508.00943/) (2025)
- [Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing](https://mlsanity.com/paper/2512.09882/) (2025)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [LLMs Outperform Experts on Challenging Biology Benchmarks](https://mlsanity.com/paper/2505.06108/) (2025)
- [Benchmarking Misuse Mitigation Against Covert Adversaries](https://mlsanity.com/paper/2506.06414/) (2025)
- [D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models](https://mlsanity.com/paper/2509.17938/) (2025)
- [Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology](https://mlsanity.com/paper/2602.16703/) (2026)
- [ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity](https://mlsanity.com/paper/2606.11150/) (2026)
- [Access Controls Will Solve the Dual-Use Dilemma](https://mlsanity.com/paper/2505.09341/) (2025)
- [An Independent Safety Evaluation of Kimi K2.5](https://mlsanity.com/paper/2604.03121/) (2026)
- [Exploration Hacking: Can LLMs Learn to Resist RL Training?](https://mlsanity.com/paper/2604.28182/) (2026)
- [Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models](https://mlsanity.com/paper/2510.27629/) (2025)
- [Jailbroken Frontier Models Retain Their Capabilities](https://mlsanity.com/paper/2605.00267/) (2026)
- [RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts](https://mlsanity.com/paper/2605.21545/) (2026)
- [Muse Spark Safety & Preparedness Report](https://mlsanity.com/paper/2606.12429/) (2026)
- [Measuring Biological Capabilities and Risks of AI Agents](https://mlsanity.com/paper/2606.19899/) (2026)
- [Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistants](https://mlsanity.com/paper/2607.13039/) (2026)
- [AI Security Leaderboard: Methodology, Results and Minimal Standard](https://mlsanity.com/paper/2608.03070/) (2026)
- [Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools](https://mlsanity.com/paper/2609.05818/) (2026)
- [Low-N Protein Activity Optimization with FolDE](https://mlsanity.com/paper/2510.24053/) (2025)
- [Risk Reporting for Developers' Internal AI Model Use](https://mlsanity.com/paper/2604.24966/) (2026)
- [Code World Model Preparedness Report](https://mlsanity.com/paper/2605.00932/) (2026)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)
- [BioSecBench-Refusal: A paired metric for performance and alignment in agentic biosecurity risk assessment](https://mlsanity.com/paper/2607.05462/) (2026)
- [Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance](https://mlsanity.com/paper/2609.16213/) (2026)

## Abstract (arXiv)

We present the Virology Capabilities Test (VCT), a large language model (LLM) benchmark that measures the capability to troubleshoot complex virology laboratory protocols. Constructed from the inputs of dozens of PhD-level expert virologists, VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories. VCT is difficult: expert virologists with access to the internet score an average of 22.1% on questions specifically in their sub-areas of expertise. However, the most performant LLM, OpenAI's o3, reaches 43.8% accuracy, outperforming 94% of expert virologists even within their sub-areas of specialization. The ability to provide expert-level virology troubleshooting is inherently dual-use: it is useful for beneficial research, but it can also be misused. Therefore, the fact that publicly available models outperform virologists on VCT raises pressing governance considerations. We propose that the capability of LLMs to provide expert-level troubleshooting of dual-use virology work should be integrated into existing frameworks for handling dual-use technologies in the life sciences.