Data to 14 Oct 2026

UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning

arXiv 2305.14761 ↗First posted 24 May 2023cs.CLcited by 160

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Charts are very popular for analyzing data, visualizing key insights and answering complex reasoning questions about data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently such as chart question answering and chart summarization. However, most of the methods that solve these tasks use pretraining on language or vision-language tasks that do not attempt to explicitly model the structure of the charts (e.g., how data is visually encoded and how chart elements are related to each other). To address this, we first build a large corpus of charts covering a wide variety of topics and visual styles. We then present UniChart, a pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual elements of charts and then uses a chart-grounded text decoder to generate the expected output in natural language. We propose several chart-specific pretraining tasks that include: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart understanding and reasoning skills. We find that pretraining the model on a large corpus with chart-specific low- and high-level tasks followed by finetuning on three down-streaming tasks results in state-of-the-art performance on three downstream tasks.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 125 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

74 of 125 references.

Closest ancestors · 1

Same direction, newest first

2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec

Other references · 73

2015Deep Residual Learning for Image Recognitioncited by 52,096
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2022Training language models to follow instructions with human feedbackcited by 15,971
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864

38 more

2022Scaling Instruction-Finetuned Language Modelscited by 3,024
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
2014Show and Tell: A Neural Image Caption Generatorcited by 879
2016A Diagram Is Worth A Dozen Imagescited by 804
2022PAL: Program-aided Language Modelscited by 652
2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
2021InfographicVQAcited by 437
2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
2021OCR-free Document Understanding Transformercited by 407
2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
2023GPTScore: Evaluate as You Desirecited by 361
2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
2022News Summarization and Evaluation in the Era of GPT-3cited by 327
2019PlotQA: Reasoning over Scientific Plotscited by 321
2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
2019Character Region Awareness for Text Detectioncited by 278
2022Is GPT-3 a Good Data Annotator?cited by 208
2019Neural Text Summarization: A Critical Evaluationcited by 189
2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 57; 2025: 50; 2026: 44.

160 papers.

Sort:

By area:

2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
2024NVILA: Efficient Frontier Visual Language Modelscited by 209
2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
2025Baichuan-Omni-1.5 Technical Reportcited by 111
2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
2024Best Practices and Lessons Learned on Synthetic Datacited by 106
2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98

38 more

2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 92
2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 89
2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
2025FineVision: Open Data Is All You Needcited by 53
2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 52
2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 34
2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
2024How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?cited by 27
2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 26
2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
2024Generative AI for Visualization: State of the Art and Future Directionscited by 23
2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
2024ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarizationcited by 19
2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 17
2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 14
2024ChartReformer: Natural Language-Driven Chart Image Editingcited by 13

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way