Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
Not on the map. Why?
Abstract (arXiv)
Chart understanding is a quintessential information fusion task, requiring the seamless integration of graphical and textual data to extract meaning. The advent of Multimodal Large Language Models (MLLMs) has revolutionized this domain, yet the landscape of MLLM-based chart analysis remains fragmented and lacks systematic organization. This survey provides a comprehensive roadmap of this nascent frontier by structuring the domain's core components. We begin by analyzing the fundamental challenges of fusing visual and linguistic information in charts. We then categorize downstream tasks and datasets, introducing a novel taxonomy of canonical and non-canonical benchmarks to highlight the field's expanding scope. Subsequently, we present a comprehensive evolution of methodologies, tracing the progression from classic deep learning techniques to state-of-the-art MLLM paradigms that leverage sophisticated fusion strategies. By critically examining the limitations of current models, particularly their perceptual and reasoning deficits, we identify promising future directions, including advanced alignment techniques and reinforcement learning for cognitive enhancement. This survey aims to equip researchers and practitioners with a structured understanding of how MLLMs are transforming chart information fusion and to catalyze progress toward more robust and reliable systems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 173 of its 238 references that are in the map. First posted on arXiv 8 Feb 2026.
Builds on
173 of 238 references.
References · 173
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
38 more
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Segment Anythingcited by 8,234
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Visual Instruction Tuningcited by 7,869
- 2017Mask R-CNNcited by 7,277
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2020Learning to summarize from human feedbackcited by 2,588
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2019Towards VQA Models That Can Readcited by 1,718
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
- 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
- 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
- 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
50 most cited of 173 All 173 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
- 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
- 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts
- 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
- 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
- 2025START: Spatial and Textual Learning for Chart Understanding
- 2026CharTool: Tool-Integrated Visual Reasoning for Chart Understanding
- 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning