mlsanityData to 14 Oct 2026

DePlot: One-shot visual language reasoning by plot-to-table translation

arXiv 2212.10505 ↗First posted 20 Dec 2022cs.CL cs.AI cs.CVcited by 160

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 51 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

30 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2022PAL: Program-aided Language Modelscited by 652
15 more
  1. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  3. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  5. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  6. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  7. 2022Binding Language Models in Symbolic Languagescited by 228
  8. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  9. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  10. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
  11. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  12. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  13. 2022Table-To-Text generation and pre-training with TabT5cited by 20
  14. 2021Classification-Regression for Chart Comprehensioncited by 11
  15. 2022TaTa: A Multilingual Table-to-Text Dataset for African Languagescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 40; 2025: 39; 2026: 62.

160 papers.

Sort:

By area:

  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  2. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  3. 2023Challenges and Applications of Large Language Modelscited by 254
  4. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  5. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  6. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  7. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  8. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  9. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  10. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  11. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  12. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
38 more
  1. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  2. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  3. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 89
  4. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  5. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  6. 2023A Survey of Reasoning with Foundation Modelscited by 70
  7. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  8. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  9. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  10. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  11. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
  12. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
  13. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  14. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  15. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  16. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 32
  17. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  18. 2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 26
  19. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24
  20. 2026PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Trainingcited by 21
  21. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  22. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  23. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  24. 2023StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understandingcited by 19
  25. 2024ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarizationcited by 19
  26. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 19
  27. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 19
  28. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 19
  29. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  30. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
  31. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  32. 2024ChartReformer: Natural Language-Driven Chart Image Editingcited by 13
  33. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  34. 2023DOMINO: A Dual-System for Multi-step Visual Language Reasoningcited by 12
  35. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 12
  36. 2024Can LLMs Understand Computer Networks? Towards a Virtual System Administratorcited by 11
  37. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  38. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  2. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMs
  3. 2026ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
  4. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  5. 2025RADAR: A Reasoning-Guided Attribution Framework for Explainable Visual Data Analysis
  6. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly
  7. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Images
  8. 2026Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts