mlsanityData to 7 Oct 2026

Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models

arXiv 2403.00231 ↗First posted 1 Mar 2024cs.CV cs.CLcited by 138

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap, we introduce ArXivQA, a question-answering dataset generated by prompting GPT-4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs' mathematical reasoning capabilities, achieving a 10.4% absolute accuracy gain on a multimodal mathematical reasoning benchmark. Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs. Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domain-specific training yields substantial performance gains. Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 63 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

32 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023CapsFusion: Rethinking Image-Text Data at Scale31 Oct

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
19 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  8. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  9. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  10. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  11. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  12. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  13. 2023The Semantic Scholar Open Data Platformcited by 109
  14. 2019On the Use of ArXiv as a Datasetcited by 101
  15. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  16. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  17. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  18. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  19. 2019Figure Captioning with Reasoning and Sequence-Level Trainingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 60; 2026: 41.

138 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  6. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  7. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  8. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  9. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  10. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
38 more
  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  2. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  3. 2025Thyme: Think Beyond Imagescited by 160
  4. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  5. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  6. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  7. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  8. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  9. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  10. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  11. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  12. 2024Reconstructive Visual Instruction Tuningcited by 54
  13. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  14. 2025FineVision: Open Data Is All You Needcited by 51
  15. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  16. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 47
  17. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  18. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  19. 2024Baichuan-Omni Technical Reportcited by 42
  20. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  21. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  22. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  23. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  24. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  25. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  26. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  27. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  28. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  29. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28
  30. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  31. 2024VILA2: VILA Augmented VILAcited by 25
  32. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  33. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  34. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  35. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
  36. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  37. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  38. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
  2. 2025Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling
  3. 2025mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models
  4. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models
  5. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
  6. 2025Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
  7. 2025A Simple Data Augmentation Strategy for Text-in-Image Scientific VQA
  8. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models