mlsanityData to 7 Oct 2026

mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model

arXiv 2311.18248 ↗First posted 30 Nov 2023cs.MM cs.CLcited by 31

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recently, the strong text creation ability of Large Language Models(LLMs) has given rise to many tools for assisting paper reading or even writing. However, the weak diagram analysis abilities of LLMs or Multimodal LLMs greatly limit their application scenarios, especially for scientific academic paper writing. In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. By aligning diagrams in the paper with related paragraphs, we construct professional diagram analysis samples for training and evaluation. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes. Besides, to better align the copilot with the user's intention, we introduce the `outline' as the control signal, which could be directly given by the user or revised based on auto-generated ones. Comprehensive experiments with a state-of-the-art Mumtimodal LLM demonstrate that training on our dataset shows stronger scientific diagram understanding performance, including diagram captioning, diagram analysis, and outline recommendation. The dataset, code, and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/PaperOwl.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 55 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

34 of 55 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
22 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  4. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  5. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  6. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  7. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  8. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  9. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  10. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  11. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  12. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  13. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  14. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  15. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  16. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  17. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  18. 2018A Web-scale system for scientific knowledge explorationcited by 39
  19. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMscited by 25
  20. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  21. 2021Enhancing Scientific Papers Summarization with Citation Graphcited by 14
  22. 2023MPMQA: Multimodal Question Answering on Product Manualscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 5; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  6. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  7. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  8. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  9. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  12. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
19 more
  1. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  2. 2024The Evolution of Multimodal Model Architecturescited by 21
  3. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  4. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  6. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  7. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understandingcited by 12
  8. 2024ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detectioncited by 11
  9. 2024VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Modelscited by 8
  10. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoderscited by 7
  11. 2024INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Modelcited by 4
  12. 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understandingcited by 4
  13. 2024MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysiscited by 3
  14. 2025Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligencecited by 2
  15. 2024Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Modelscited by 1
  16. 2024AceParse: A Comprehensive Dataset with Diverse Structured Texts for Academic Literature Parsingcited by 1
  17. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  18. 2026Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offscited by 0
  19. 2026SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
  2. 2025Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots
  3. 2025Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts
  4. 2026Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
  5. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
  6. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation
  7. 2025Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models
  8. 2026SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models