# MuSciClaims: Multimodal Scientific Claim Verification

arXiv: https://arxiv.org/abs/2506.04585 · HTML: https://mlsanity.com/paper/2506.04585/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning](https://mlsanity.com/paper/2311.10774/) (2023)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [SciFact-Open: Towards open-domain scientific claim verification](https://mlsanity.com/paper/2210.13777/) (2022)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs](https://mlsanity.com/paper/2308.03349/) (2023)
- [SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)](https://mlsanity.com/paper/2105.13995/) (2021)
- [Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking](https://mlsanity.com/paper/2301.11843/) (2023)
- [CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweets](https://mlsanity.com/paper/2204.12164/) (2022)
- [ChartCheck: Explainable Fact-Checking over Real-World Chart Images](https://mlsanity.com/paper/2311.07453/) (2023)
- [MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems](https://mlsanity.com/paper/2410.14179/) (2024)
- [SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers](https://mlsanity.com/paper/2411.05338/) (2024)
- [Evaluating Paraphrastic Robustness in Textual Entailment Models](https://mlsanity.com/paper/2306.16722/) (2023)

## Built on by (most cited first, 8 of 8)

- [SciClaimEval: Cross-modal Claim Verification in Scientific Papers](https://mlsanity.com/paper/2602.07621/) (2026)
- [Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts](https://mlsanity.com/paper/2511.10075/) (2025)
- [M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency](https://mlsanity.com/paper/2604.01306/) (2026)
- [When Verification Fails: How Compositionally Infeasible Claims Escape Rejection](https://mlsanity.com/paper/2604.10990/) (2026)
- [Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification](https://mlsanity.com/paper/2606.01679/) (2026)
- [ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.16131/) (2026)
- [Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing](https://mlsanity.com/paper/2608.07437/) (2026)
- [MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places](https://mlsanity.com/paper/2608.28384/) (2026)

## Abstract (arXiv)

Assessing scientific claims requires identifying, extracting, and reasoning with multimodal data expressed in information-rich figures in scientific literature. Despite the large body of work in scientific QA, figure captioning, and other multimodal reasoning tasks over chart-based data, there are no readily usable multimodal benchmarks that directly test claim verification abilities. To remedy this gap, we introduce a new benchmark MuSciClaims accompanied by diagnostics tasks. We automatically extract supported claims from scientific articles, which we manually perturb to produce contradicted claims. The perturbations are designed to test for a specific set of claim verification capabilities. We also introduce a suite of diagnostic tasks that help understand model failures. Our results show most vision-language models are poor (~0.3-0.5 F1), with even the best model only achieving 0.72 F1. They are also biased towards judging claims as supported, likely misunderstanding nuanced perturbations within the claims. Our diagnostics show models are bad at localizing correct evidence within figures, struggle with aggregating information across modalities, and often fail to understand basic components of the figure.