mlsanityData to 7 Oct 2026

SFBench: The SciFy Scientific Feasibility Benchmark

arXiv 2606.29630 ↗First posted 28 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 68 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

46 of 68 references.

References · 46

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
34 more
  1. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  5. 2025Humanity's Last Examcited by 467
  6. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  7. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 375
  8. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  9. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  10. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  11. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 174
  12. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 171
  13. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  14. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  15. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  16. 2021COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemiccited by 100
  17. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  18. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 94
  19. 2024SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoningcited by 90
  20. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 67
  21. 2024Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent Systemcited by 63
  22. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 47
  23. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 42
  24. 2025Evaluating Large Language Models in Scientific Discoverycited by 38
  25. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 32
  26. 2024ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictorcited by 21
  27. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  28. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  29. 2025CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?cited by 11
  30. 2024A Guide to Misinformation Detection Data and Evaluationcited by 10
  31. 2024WithdrarXiv: A Large-Scale Dataset for Retraction Studycited by 9
  32. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  33. 2025MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoningcited by 6
  34. 2025MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidancecited by 3

Built on by

Not cited yet.