# SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification

arXiv: https://arxiv.org/abs/2506.15569 · HTML: https://mlsanity.com/paper/2506.15569/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [SciFact-Open: Towards open-domain scientific claim verification](https://mlsanity.com/paper/2210.13777/) (2022)
- [INFOTABS: Inference on Tables as Semi-structured Data](https://mlsanity.com/paper/2005.06117/) (2020)
- [SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables](https://mlsanity.com/paper/2305.13186/) (2023)
- [Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs](https://mlsanity.com/paper/2402.12424/) (2024)
- [AmbiFC: Fact-Checking Ambiguous Claims with Evidence](https://mlsanity.com/paper/2104.00640/) (2021)
- [M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2411.04075/) (2024)
- [ChartCheck: Explainable Fact-Checking over Real-World Chart Images](https://mlsanity.com/paper/2311.07453/) (2023)
- [Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation](https://mlsanity.com/paper/2309.08793/) (2023)
- [FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents](https://mlsanity.com/paper/2411.05764/) (2024)

## Built on by (most cited first, 17 of 17)

- [Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers](https://mlsanity.com/paper/2507.10787/) (2025)
- [SciClaimEval: Cross-modal Claim Verification in Scientific Papers](https://mlsanity.com/paper/2602.07621/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [SciMDR: Advancing Scientific Multimodal Document Reasoning](https://mlsanity.com/paper/2603.12249/) (2026)
- [Analyzing LLM Instruction Optimization for Tabular Fact Verification](https://mlsanity.com/paper/2602.17937/) (2026)
- [M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency](https://mlsanity.com/paper/2604.01306/) (2026)
- [CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text](https://mlsanity.com/paper/2605.27700/) (2026)
- [Are Two LLMs Better Than One? A Student-Teacher Dual-Head LLMs Architecture for Pharmaceutical Content Optimization](https://mlsanity.com/paper/2602.11957/) (2026)
- [When Verification Fails: How Compositionally Infeasible Claims Escape Rejection](https://mlsanity.com/paper/2604.10990/) (2026)
- [DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation](https://mlsanity.com/paper/2605.27710/) (2026)
- [Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification](https://mlsanity.com/paper/2606.01679/) (2026)
- [ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.16131/) (2026)
- [PEARL: Auditable Repair for Scientific Reasoning Graph Extraction](https://mlsanity.com/paper/2607.17917/) (2026)
- [XAI-Refine: An Automated Explanation-Knowledge Loop for Brain-Age Prediction](https://mlsanity.com/paper/2609.09388/) (2026)
- [BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction](https://mlsanity.com/paper/2609.24921/) (2026)
- [When Scientific Contradictions Are Lost in Translation](https://mlsanity.com/paper/2609.38621/) (2026)
- [How Robust Is Multimodal Claim Verification to LLM Rewriting?](https://mlsanity.com/paper/2610.02841/) (2026)

## Abstract (arXiv)

We introduce SciVer, the first benchmark specifically designed to evaluate the ability of foundation models to verify claims within a multimodal scientific context. SciVer consists of 3,000 expert-annotated examples over 1,113 scientific papers, covering four subsets, each representing a common reasoning type in multimodal scientific claim verification. To enable fine-grained evaluation, each example includes expert-annotated supporting evidence. We assess the performance of 21 state-of-the-art multimodal foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and Qwen2.5-VL. Our experiment reveals a substantial performance gap between these models and human experts on SciVer. Through an in-depth analysis of retrieval-augmented generation (RAG), and human-conducted error evaluations, we identify critical limitations in current open-source models, offering key insights to advance models' comprehension and reasoning in multimodal scientific literature tasks.