mlsanityData to 7 Oct 2026

TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning

arXiv 2404.16635 ↗First posted 25 Apr 2024cs.CVcited by 51

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Charts are important for presenting and explaining complex data relationships. Recently, multimodal large language models (MLLMs) have shown remarkable capabilities in various chart understanding tasks. However, the sheer size of these models in terms of parameters and computational requirements limits their use in resource-constrained environments. In this paper, we present TinyChart, an efficient MLLM for chart understanding with only 3B parameters. TinyChart overcomes two key challenges in efficient chart understanding: (1) reduce the burden of learning numerical computations through a Program-of-Thoughts (PoT) learning strategy, which trains the model to generate Python programs for numerical calculations, and (2) reduce lengthy vision feature sequences produced by the vision transformer for high-resolution images through a Vision Token Merging module, which gradually merges most similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart achieves SOTA performance on a variety of chart understanding benchmarks including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It outperforms several chart understanding MLLM with up to 13B parameters such as ChartLlama and ChartAst, and close-sourced general-purpose MLLM GPT-4V on ChartQA. It also demonstrates its superior efficiency with higher throughput during inference due to a smaller model scale and more efficient vision encoding. Our code and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/TinyChart.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 61 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

34 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning14 Mar
  2. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb

Other references · 32

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  10. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
20 more
  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  3. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  4. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  6. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  7. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  8. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  9. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  10. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  11. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  12. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  13. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  14. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  15. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  16. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  17. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  18. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  19. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 31
  20. 2024Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 15.

51 papers.

Sort:

By area:

  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  3. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  4. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 84
  5. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  6. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  7. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  8. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understandingcited by 13
  9. 2025BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuningcited by 13
  10. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 10
  11. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?cited by 10
  12. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
38 more
  1. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  2. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 8
  3. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
  4. 2024VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoningcited by 6
  5. 2025Protecting multimodal large language models against misleading visualizationscited by 6
  6. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  7. 2024Towards Unlocking Insights from Logbooks Using AIcited by 5
  8. 2025SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understandingcited by 5
  9. 2025The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Modelscited by 5
  10. 2024ChartAdapter: Large Vision-Language Model for Chart Summarizationcited by 4
  11. 2025Is this chart lying to me? Automating the detection of misleading visualizationscited by 4
  12. 2025START: Spatial and Textual Learning for Chart Understandingcited by 4
  13. 2025ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoningcited by 4
  14. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  15. 2025Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languagescited by 3
  16. 2025Do MLLMs Really Understand the Charts?cited by 3
  17. 2025Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Datacited by 2
  18. 2025Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Frameworkcited by 2
  19. 2026ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Chartscited by 2
  20. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  21. 2024EVLM: Self-Reflective Multimodal Reasoning for Cross-Dimensional Visual Editingcited by 1
  22. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  23. 2025ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehensioncited by 1
  24. 2025Towards Lossless Ultimate Vision Token Compression for VLMscited by 1
  25. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusioncited by 1
  26. 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understandingcited by 1
  27. 2025Coling-UniA at SciVQA 2025: Few-Shot Example Retrieval and Confidence-Informed Ensembling for Multimodal Large Language Modelscited by 0
  28. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  29. 2026ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagerycited by 0
  30. 2026CharTool: Tool-Integrated Visual Reasoning for Chart Understandingcited by 0
  31. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  32. 2026Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translationcited by 0
  33. 2026PlotPick: AI-powered batch extraction of numerical data from scientific figurescited by 0
  34. 2026EpiCurveBench: Evaluating VLMs on Epidemic Curve Digitizationcited by 0
  35. 2026Self-Ensembling Vision-Language Models for Chart Data Extractioncited by 0
  36. 2026REChart: Reasoning-Efficient Chart Editing with Large Reasoning Modelscited by 0
  37. 2026TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Modelscited by 0
  38. 2026LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventionscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
  2. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  3. 2026ChartComplete: A Taxonomy-based Inclusive Chart Dataset
  4. 2024AskChart: Universal Chart Understanding through Textual Enhancement
  5. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  6. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
  7. 2024On Pre-training of Multimodal Language Models Customized for Chart Understanding
  8. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts