mlsanityData to 7 Oct 2026

SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation

arXiv 2405.08807 ↗First posted 14 May 2024cs.CVcited by 47

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) have proven flexible and generalisable across many tasks and fields. Although they have strong potential to aid scientific research, their capabilities in this domain are not well characterised. A key aspect of scientific research is the ability to understand and interpret figures, which serve as a rich, compressed source of complex information. In this work, we present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories. The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control. We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark. Finally, we investigate the alignment and reasoning faithfulness of the LMMs on augmented question sets from our benchmark. We release SciFIBench to encourage progress in this domain.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 74 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

54 of 74 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  12. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
1 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
29 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  5. 2024The Faiss librarycited by 370
  6. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  7. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  8. 2023Demystifying CLIP Datacited by 265
  9. 2023Data Filtering Networkscited by 247
  10. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  11. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  12. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  13. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  14. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  15. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  16. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 85
  17. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  18. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  19. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  20. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  21. 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray imagescited by 37
  22. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  23. 2023LLMs for Science: Usage for Code Generation and Data Analysiscited by 34
  24. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  25. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  26. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 22
  27. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  28. 2023ACL-Fig: A Dataset for Scientific Figure Classificationcited by 10
  29. 2023FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedbackcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 19; 2026: 20.

47 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  3. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  4. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  7. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  8. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
  9. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  10. 2025SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Modelcited by 18
  11. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  12. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
35 more
  1. 2025MatTools: Benchmarking Large Language Models for Materials Science Toolscited by 11
  2. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  3. 2025RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluationcited by 10
  4. 2025MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMscited by 9
  5. 2026QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understandingcited by 9
  6. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 8
  7. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  8. 2024MMR: Evaluating Reading Ability of Large Multimodal Modelscited by 6
  9. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  10. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 4
  11. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  12. 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understandingcited by 4
  13. 2026SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detectioncited by 4
  14. 2024GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Modelscited by 3
  15. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
  16. 2026SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literaturecited by 3
  17. 2026GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Modelscited by 2
  18. 2026MemoNoveltyAgent: A Historical Research Memory-Aware Agent Workflow for Paper Novelty Assessmentcited by 1
  19. 2026Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agentscited by 1
  20. 2026Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discoverycited by 1
  21. 2026OmniScientist: An Omni-Modal Omni-Discipline AI Scientistcited by 1
  22. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  23. 2025VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answeringcited by 0
  24. 2026S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understandingcited by 0
  25. 2025AI-Assisted Authoring for Transparent, Data-Driven Documentscited by 0
  26. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  27. 2026When Does Multimodal AI Help? Diagnostic Complementarity of Vision-Language Models and CNNs for Spectrum Management in Satellite-Terrestrial Networkscited by 0
  28. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0
  29. 2026SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Imagescited by 0
  30. 2026PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Modelscited by 0
  31. 2026SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidencecited by 0
  32. 2026SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Contextcited by 0
  33. 2026How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figurescited by 0
  34. 2026BusMA: A Bus Communication Substrate for Multi-Agent Systemscited by 0
  35. 2026PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slidescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  2. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  3. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models
  4. 2025SCI-Reason: A Dataset with Chain-of-Thought Rationales for Complex Multimodal Reasoning in Academic Areas
  5. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
  6. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  7. 2025Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling
  8. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models