mlsanityData to 14 Oct 2026

PlotQA: Reasoning over Scientific Plots

arXiv 1909.00997 ↗First posted 3 Sep 2019cs.CV cs.AI cs.CLcited by 321

Not on the map. Why?

Cited by 143 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing synthetic datasets (FigureQA, DVQA) for reasoning over plots do not contain variability in data labels, real-valued data, or complex reasoning questions. Consequently, proposed models for these datasets do not fully address the challenge of reasoning over plots. In particular, they assume that the answer comes either from a small fixed size vocabulary or from a bounding box within the image. However, in practice, this is an unrealistic assumption because many questions require reasoning and thus have real-valued answers which appear neither in a small fixed size vocabulary nor in the image. In this work, we aim to bridge this gap between existing datasets and real-world plots. Specifically, we propose PlotQA with 28.9 million question-answer pairs over 224,377 plots on data from real-world sources and questions based on crowd-sourced question templates. Further, 80.76% of the out-of-vocabulary (OOV) questions in PlotQA have answers that are not in a fixed vocabulary. Analysis of existing models on PlotQA reveals that they cannot deal with OOV questions: their overall accuracy on our dataset is in single digits. This is not surprising given that these models were not designed for such questions. As a step towards a more holistic model which can address fixed vocabulary as well as OOV questions, we propose a hybrid approach: Specific questions are answered by choosing the answer from a fixed vocabulary or by extracting it from a predicted bounding box in the plot, while other questions are answered with a table question-answering engine which is fed with a structured table generated by detecting visual elements from the image. On the existing DVQA dataset, our model has an accuracy of 58%, significantly improving on the highest reported accuracy of 46%. On PlotQA, our model has an accuracy of 22.52%, which is significantly better than state of the art models.

Key facts. Cited by 321 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 27; 2024: 78; 2025: 95; 2026: 109.

321 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  7. 2024What matters when building vision-language models?cited by 326
  8. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  9. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  10. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  11. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  12. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
38 more
  1. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  2. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  3. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  6. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  7. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  8. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  9. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  10. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  11. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  12. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  13. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  14. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
  15. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  16. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  17. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  18. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  19. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  20. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  21. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 108
  22. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  23. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  24. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  25. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  26. 2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 85
  27. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  28. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  29. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  30. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 76
  31. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  32. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  33. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  34. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  35. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  36. 2022OpenCQA: Open-ended Question Answering with Chartscited by 61
  37. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  38. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 58

50 most cited of 321 All 321 (JSON)