mlsanityData to 14 Oct 2026

SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables

arXiv 2305.13186 ↗First posted 22 May 2023cs.CL cs.AIcited by 51

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publications and 2) require compositional reasoning for verification. The claims are paired with evidence-containing scientific tables annotated with labels. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models, including table-based pretraining models and large language models. All models except GPT-4 achieved performance barely above random guessing. Popular prompting techniques, such as Chain-of-Thought, do not achieve much performance gains on SCITAB. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning. Our codes and data are publicly available at https://github.com/XinyuanLu00/SciTab.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 63 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

43 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning31 Jan

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
30 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  4. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  5. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  6. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  7. 2021A Survey on Automated Fact-Checkingcited by 378
  8. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  9. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  10. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  11. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  12. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  13. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 167
  14. 2019Towards Debiasing Fact Verification Modelscited by 143
  15. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  16. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  17. 2021COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemiccited by 100
  18. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  19. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
  20. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 86
  21. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  22. 2022PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-trainingcited by 41
  23. 2021AmbiFC: Fact-Checking Ambiguous Claims with Evidencecited by 38
  24. 2021SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)cited by 35
  25. 2020Program Enhanced Fact Verification with Verbalization and Graph Attention Networkcited by 29
  26. 2022CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweetscited by 29
  27. 2020LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Networkcited by 26
  28. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14
  29. 2022Table-based Fact Verification with Self-adaptive Mixture of Expertscited by 12
  30. 2020SciWING -- A Software Toolkit for Scientific Document Processingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 20; 2026: 16.

51 papers.

Sort:

By area:

  1. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  2. 2023QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checkingcited by 26
  3. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 24
  4. 2023Effective Distillation of Table-based Reasoning Ability from LLMscited by 23
  5. 2024ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Contextcited by 23
  6. 2024FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documentscited by 19
  7. 2025SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verificationcited by 17
  8. 2024Missci: Reconstructing Fallacies in Misrepresented Sciencecited by 14
  9. 2024PoTable: Towards Systematic Thinking via Plan-then-Execute Stage Reasoning on Tablescited by 14
  10. 2024Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answeringcited by 13
  11. 2025Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tablescited by 13
  12. 2025CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?cited by 12
38 more
  1. 2025RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysiscited by 12
  2. 2026RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Modelscited by 11
  3. 2024TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoningcited by 10
  4. 2024SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Typescited by 10
  5. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  6. 2024Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysiscited by 9
  7. 2024CHECKWHY: Causal Fact Verification via Argument Structurecited by 9
  8. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  9. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  10. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  11. 2026SciClaimEval: Cross-modal Claim Verification in Scientific Paperscited by 7
  12. 2024CoverBench: A Challenging Benchmark for Complex Claim Verificationcited by 6
  13. 2025arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generationcited by 6
  14. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  15. 2025Texts or Images? A Fine-grained Analysis on the Effectiveness of Input Representations and Models for Table Question Answeringcited by 5
  16. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  17. 2025Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluationcited by 5
  18. 2023Investigating Zero- and Few-shot Generalization in Fact Verificationcited by 4
  19. 2025Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Paperscited by 4
  20. 2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
  21. 2026ClaimDB: A Fact Verification Benchmark over Large Structured Datacited by 3
  22. 2025ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Chartscited by 2
  23. 2025TSVer: A Benchmark for Fact Verification Against Time-Series Evidencecited by 2
  24. 2026DecomposeRL: Learning to Ask Useful, Informative, and Diverse Questions for Semi-Supervised, Traceable Claim Verificationcited by 2
  25. 2025p2-TQA: A Process-based Preference Learning Framework for Self-Improving Table Question Answering Modelscited by 1
  26. 2025Batch Prompting Suppresses Overthinking Reasoning Under Constraint: How Batch Prompting Suppresses Overthinking in Reasoning Modelscited by 1
  27. 2026Analyzing LLM Instruction Optimization for Tabular Fact Verificationcited by 1
  28. 2026When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errorscited by 1
  29. 2026SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understandingcited by 1
  30. 2025ModelTables: A Corpus of Tables about Modelscited by 0
  31. 2026Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Datacited by 0
  32. 2026GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoningcited by 0
  33. 2026SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computationcited by 0
  34. 2026When Verification Fails: How Compositionally Infeasible Claims Escape Rejectioncited by 0
  35. 2026A Multi-Agent Approach for Claim Verification from Tabular Data Documentscited by 0
  36. 2026Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verificationcited by 0
  37. 2026SciR: A Controllable Benchmark for Scientific Reasoning in LLMscited by 0
  38. 2026ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learningcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Atomic Reasoning for Scientific Table Claim Verification
  2. 2026SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding
  3. 2026SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task
  4. 2025Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim rightarrow Evidence Reasoning
  5. 2026What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
  6. 2026SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature
  7. 2026Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking
  8. 2026RSAT: Structured Attribution Makes Small Language Models Faithful Table Reasoners