K-Bench: measuring model performance on real scientific agent requests
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and they lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor is defined as work a domain scientist would accept with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 63 of its 74 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
63 of 74 references.
Closest ancestors · 13
Same direction, newest first
- 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents6 Feb
- 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents10 Nov
- 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents13 Jun
- 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agents20 Feb
- 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts22 Nov
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
- 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery7 Oct
- 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark17 Sep
- 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep
- 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
- 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science19 Aug
- 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Models1 Jul
1 more
Other references · 50
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
38 more
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
- 2025Humanity's Last Examcited by 491
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
- 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
- 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
- 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
- 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
- 2021The Benchmark Lotterycited by 136
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 115
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
- 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 89
- 2025The Leaderboard Illusioncited by 66
- 2025The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideascited by 62
- 2025AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suitecited by 53
- 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
- 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 47
- 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 47
- 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 43
- 2025Evaluating Large Language Models in Scientific Discoverycited by 42
- 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
- 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 16
- 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16
- 2025Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Labcited by 12
- 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Researchcited by 8
- 2025K-Dense Analyst: Towards Fully Automated Scientific Analysiscited by 6
- 2026RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessionscited by 3
- 2026PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocolscited by 1
- 2026Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientistscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
- 2026Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
- 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
- 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Research
- 2025FML-bench: Benchmarking Machine Learning Agents for Scientific Research
- 2026Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems
- 2026Efficient Benchmarking of AI Agents
- 2026ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences