mlsanityData to 7 Oct 2026

ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild

arXiv 2407.04172 ↗First posted 4 Jul 2024cs.AI cs.CL cs.CVcited by 79

Not on the map. Why?

Cited by 41 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across 5 benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

26 of 45 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  12. 2019PlotQA: Reasoning over Scientific Plotscited by 317
14 more
  1. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  4. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  5. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  6. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  7. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  8. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  9. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  10. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  11. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  12. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  13. 2022Chart Question Answering: State of the Art and Future Directionscited by 26
  14. 2024On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 41; 2026: 28.

79 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  4. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  5. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  6. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  7. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  8. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  9. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  10. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  11. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  12. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
38 more
  1. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  2. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  3. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  4. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  5. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  6. 2025BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuningcited by 13
  7. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  8. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  9. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  10. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  11. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 10
  12. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Uglycited by 9
  13. 2025Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptationcited by 9
  14. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  15. 2026QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understandingcited by 9
  16. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 8
  17. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  18. 2025PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent Systemcited by 7
  19. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directionscited by 7
  20. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  21. 2024VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoningcited by 6
  22. 2024Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Modelscited by 6
  23. 2025Protecting multimodal large language models against misleading visualizationscited by 6
  24. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  25. 2024Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directionscited by 5
  26. 2025The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Modelscited by 5
  27. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
  28. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understandingcited by 4
  29. 2025START: Spatial and Textual Learning for Chart Understandingcited by 4
  30. 2025Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptionscited by 3
  31. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  32. 2025DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboardscited by 3
  33. 2025Do MLLMs Really Understand the Charts?cited by 3
  34. 2025ChartAB: A Benchmark for Chart Grounding & Dense Alignmentcited by 3
  35. 2026TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RLcited by 3
  36. 2025Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Datacited by 2
  37. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understandingcited by 2
  38. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  2. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information
  3. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
  4. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
  5. 2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
  6. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  7. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  8. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules