mlsanityData to 14 Oct 2026

SciFact-Open: Towards open-domain scientific claim verification

arXiv 2210.13777 ↗First posted 25 Oct 2022cs.CL cs.AIcited by 94

Not on the map. Why?

Abstract (arXiv)

While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

26 of 53 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  10. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  12. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
14 more
  1. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  2. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  3. 2021COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemiccited by 100
  4. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  5. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  6. 2020Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarizationcited by 52
  7. 2020Scientific Claim Verification with VERT5ERINIcited by 47
  8. 2020Evidence-based Factual Error Correctioncited by 44
  9. 2022BERN2: an advanced neural biomedical named entity recognition and normalization toolcited by 42
  10. 2020A Paragraph-level Multi-task Learning Model for Scientific Fact-Verificationcited by 29
  11. 2019Automatic Fact-guided Sentence Modificationcited by 28
  12. 2021Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verificationcited by 20
  13. 2021Overview and Insights from the SciVer Shared Task on Scientific Claim Verificationcited by 17
  14. 2021The Choice of Knowledge Base in Automated Claim Checkingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 16; 2025: 29; 2026: 35.

94 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  3. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  4. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  5. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109
  6. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  7. 2023Scientific Fact-Checking: A Survey of Resources and Approachescited by 54
  8. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tablescited by 51
  9. 2023Do Large Language Models Know about Facts?cited by 51
  10. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 51
  11. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
  12. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
38 more
  1. 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Taskscited by 38
  2. 2023Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Modelscited by 37
  3. 2023Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generationcited by 25
  4. 2024What Can Natural Language Processing Do for Peer Review?cited by 25
  5. 2023Decomposing Complex Queries for Tip-of-the-tongue Retrievalcited by 24
  6. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 24
  7. 2024FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documentscited by 19
  8. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  9. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  10. 2025SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verificationcited by 17
  11. 2025IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrievalcited by 16
  12. 2024Missci: Reconstructing Fallacies in Misrepresented Sciencecited by 14
  13. 2024Improving Health Question Answering with Reliable and Time-Aware Evidence Retrievalcited by 12
  14. 2025CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?cited by 12
  15. 2024What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verificationcited by 10
  16. 2024Comparing Knowledge Sources for Open-Domain Scientific Claim Verificationcited by 10
  17. 2024Can Large Language Models Detect Misinformation in Scientific News Reporting?cited by 9
  18. 2024On Evaluating Explanation Utility for Human-AI Decision Making in NLPcited by 9
  19. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  20. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  21. 2023SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examplescited by 8
  22. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  23. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  24. 2026SciClaimEval: Cross-modal Claim Verification in Scientific Paperscited by 7
  25. 2022Combating Health Misinformation in Social Media: Characterization, Detection, Intervention, and Open Issuescited by 6
  26. 2023Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciencescited by 6
  27. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  28. 2024Information Re-Organization Improves Reasoning in Large Language Modelscited by 5
  29. 2025Span-Level Hallucination Detection for LLM-Generated Answerscited by 5
  30. 2026Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectorycited by 5
  31. 2024Automated Fact-Checking of Climate Change Claims with Large Language Modelscited by 4
  32. 2025SciClaimHunt: A Large Dataset for Evidence-based Scientific Claim Verificationcited by 4
  33. 2024Robust Claim Verification Through Fact Detectioncited by 3
  34. 2025A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decompositioncited by 3
  35. 2026Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checkingcited by 3
  36. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  37. 2025NSF-SciFy: Mining the NSF Awards Database for Scientific Claimscited by 2
  38. 2025Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outscited by 2

50 most cited of 94 All 94 (JSON)