mlsanityData to 14 Oct 2026

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark

arXiv 2504.16137 ↗First posted 21 Apr 2025cs.CY cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

We present the Virology Capabilities Test (VCT), a large language model (LLM) benchmark that measures the capability to troubleshoot complex virology laboratory protocols. Constructed from the inputs of dozens of PhD-level expert virologists, VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories. VCT is difficult: expert virologists with access to the internet score an average of 22.1% on questions specifically in their sub-areas of expertise. However, the most performant LLM, OpenAI's o3, reaches 43.8% accuracy, outperforming 94% of expert virologists even within their sub-areas of specialization. The ability to provide expert-level virology troubleshooting is inherently dual-use: it is useful for beneficial research, but it can also be misused. Therefore, the fact that publicly available models outperform virologists on VCT raises pressing governance considerations. We propose that the capability of LLMs to provide expert-level troubleshooting of dual-use virology work should be integrated into existing frameworks for handling dual-use technologies in the life sciences.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

33 of 59 references.

References · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
21 more
  1. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  3. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  5. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  6. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  7. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  8. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  9. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  10. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 65
  11. 2024CRISPR-GPT for Agentic Automation of Gene-editing Experimentscited by 58
  12. 2023Artificial intelligence and biological misuse: Differentiating risks of language models and biological design toolscited by 45
  13. 2024How predictable is language model benchmark performance?cited by 37
  14. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 26
  15. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 25
  16. 2024MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMscited by 16
  17. 2024Introduction to AI Safety, Ethics, and Societycited by 15
  18. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  19. 2024Towards Risk Analysis of the Impact of AI on the Deliberate Biological Threat Landscapecited by 1
  20. 2024The whack-a-mole governance challenge for AI-enabled synthetic biology: literature review and emerging frameworkscited by 1
  21. 2025Foundation models may exhibit staged progression in novel CBRN threat disclosurecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 18.

31 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  3. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  4. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
  5. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 18
  6. 2025Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testingcited by 15
  7. 2026Shaping capabilities with token-level data filteringcited by 13
  8. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 10
  9. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  10. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  11. 2025D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Modelscited by 7
  12. 2026Measuring Mid-2025 LLM-Assistance on Novice Performance in Biologycited by 6
19 more
  1. 2026ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecuritycited by 6
  2. 2025Access Controls Will Solve the Dual-Use Dilemmacited by 3
  3. 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
  4. 2026Exploration Hacking: Can LLMs Learn to Resist RL Training?cited by 2
  5. 2025Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Modelscited by 1
  6. 2026Jailbroken Frontier Models Retain Their Capabilitiescited by 1
  7. 2026RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Promptscited by 1
  8. 2026Muse Spark Safety & Preparedness Reportcited by 1
  9. 2026Measuring Biological Capabilities and Risks of AI Agentscited by 1
  10. 2026Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistantscited by 1
  11. 2026AI Security Leaderboard: Methodology, Results and Minimal Standardcited by 1
  12. 2026Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Toolscited by 1
  13. 2025Low-N Protein Activity Optimization with FolDEcited by 0
  14. 2026Risk Reporting for Developers' Internal AI Model Usecited by 0
  15. 2026Code World Model Preparedness Reportcited by 0
  16. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  17. 2026BioSecBench-Refusal: A paired metric for performance and alignment in agentic biosecurity risk assessmentcited by 0
  18. 2026Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governancecited by 0
  19. 2026DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Designcited by 0