mlsanityData to 7 Oct 2026

MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering

arXiv 2212.09662 ↗First posted 19 Dec 2022cs.CL cs.AI cs.CVcited by 132

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models' capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 48 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

34 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2022PAL: Program-aided Language Modelscited by 635
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
21 more
  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  4. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  5. 2021OCR-free Document Understanding Transformercited by 400
  6. 2015Neural Module Networkscited by 375
  7. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  8. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  9. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  10. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  11. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  12. 2022Binding Language Models in Symbolic Languagescited by 224
  13. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  14. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  15. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  16. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  17. 2020Understanding tables with intermediate pre-trainingcited by 68
  18. 2022Reasoning Like Program Executorscited by 45
  19. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  20. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  21. 2021Classification-Regression for Chart Comprehensioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 23; 2024: 42; 2025: 26; 2026: 40.

132 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  4. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  5. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  6. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  7. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  8. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  9. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  10. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  11. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  12. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
38 more
  1. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  2. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  3. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  4. 2023A Survey of Reasoning with Foundation Modelscited by 70
  5. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  6. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  7. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  8. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  9. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  10. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  11. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  12. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  13. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  14. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  15. 2024How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?cited by 27
  16. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  17. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  18. 2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 25
  19. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24
  20. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  21. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  22. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  23. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  24. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  25. 2024ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarizationcited by 19
  26. 2023StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understandingcited by 18
  27. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 18
  28. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
  29. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  30. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answeringcited by 14
  31. 2024ChartReformer: Natural Language-Driven Chart Image Editingcited by 13
  32. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  33. 2023DOMINO: A Dual-System for Multi-step Visual Language Reasoningcited by 12
  34. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  35. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  36. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  37. 2023Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarizationcited by 10
  38. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 10

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
  2. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly
  3. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  4. 2025End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models
  5. 2025ChartAB: A Benchmark for Chart Grounding & Dense Alignment
  6. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding
  7. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
  8. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information