mlsanityData to 7 Oct 2026

From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models

arXiv 2403.12027 ↗First posted 18 Mar 2024cs.CL cs.AI cs.CVcited by 52

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Automatic chart understanding has witnessed significant advancements with the rise of large foundation models in recent years. Foundation models, such as large language models, have revolutionized various natural language processing tasks and are increasingly being applied to chart understanding tasks. This survey paper provides a comprehensive overview of the recent developments, challenges, and future directions in chart understanding within the context of these foundation models. We review fundamental building blocks crucial for studying chart understanding tasks. Additionally, we explore various tasks and their evaluation metrics and sources of both charts and textual inputs. Various modeling strategies are then examined, encompassing both classification-based and generation-based approaches, along with tool augmentation techniques that enhance chart understanding performance. Furthermore, we discuss the state-of-the-art performance of each task and discuss how we can improve the performance. Challenges and future directions are addressed, highlighting the importance of several topics, such as domain-specific charts, lack of efforts in developing evaluation metrics, and agent-oriented settings. This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis, providing valuable insights and directions for future research in chart understanding leveraging large foundation models. The studies mentioned in this paper, along with emerging new research, will be continually updated at: https://github.com/khuangaf/Awesome-Chart-Understanding.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 107 of its 146 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

107 of 146 references.

References · 107

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2016Densely Connected Convolutional Networkscited by 6,691
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  13. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  14. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  15. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  18. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  19. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  20. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  21. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  22. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  23. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  24. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  25. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  26. 2021OCR-free Document Understanding Transformercited by 400
  27. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  28. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  29. 2017A simple neural network module for relational reasoningcited by 345
  30. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  31. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  32. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  33. 2023Tool Learning with Foundation Modelscited by 302
  34. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  35. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  36. 2019Character Region Awareness for Text Detectioncited by 273
  37. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  38. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 27; 2026: 14.

52 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  3. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
  4. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  5. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  6. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  7. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  8. 2024Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Taskscited by 30
  9. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  10. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 23
  11. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Modelscited by 16
  12. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
38 more
  1. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  2. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 10
  3. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  4. 2025On the Perception Bottleneck of VLMs for Chart Understandingcited by 9
  5. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Uglycited by 9
  6. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  7. 2025ChartEditor: A Reinforcement Learning Framework for Robust Chart Editingcited by 9
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  9. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  10. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forwardcited by 7
  11. 2024Data Analysis in the Era of Generative AIcited by 6
  12. 2025Protecting multimodal large language models against misleading visualizationscited by 6
  13. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answeringcited by 6
  14. 2024CHARTOM: A Visual Theory-of-Mind Benchmark for LLMs on Misleading Chartscited by 5
  15. 2025CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Modelscited by 5
  16. 2025InfoDet: A Dataset for Infographic Element Detectioncited by 4
  17. 2025Towards the Autonomous Optimization of Urban Logistics: Training Generative AI with Scientific Tools via Agentic Digital Twins and Model Context Protocolcited by 4
  18. 2025ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoningcited by 4
  19. 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Chartscited by 3
  20. 2026Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewardscited by 3
  21. 2025ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Chartscited by 2
  22. 2025From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Textcited by 2
  23. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 2
  24. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Modelscited by 2
  25. 2026ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinementcited by 2
  26. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  27. 2024Empowering Cognitive Digital Twins with Generative Foundation Models: Developing a Low-Carbon Integrated Freight Transportation Systemcited by 1
  28. 2025Personalized Recommendation Models in Federated Settings: A Surveycited by 1
  29. 2025Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discoverycited by 1
  30. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  31. 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understandingcited by 1
  32. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  33. 2025CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generationcited by 0
  34. 2026ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generationcited by 0
  35. 2026MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrievalcited by 0
  36. 2026Large language model-enabled automated data extraction for concrete materials informaticscited by 0
  37. 2026Progressive Refinement: An Iterative Pseudo-Labeling Approach for Mandarin-English Code-Switching ASRcited by 0
  38. 2026Chart Deception in Vision-Language Models: From Vulnerability to Mitigationcited by 0

50 most cited of 52 All 52 (JSON)