# SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials

arXiv: https://arxiv.org/abs/2405.00021 · HTML: https://mlsanity.com/paper/2405.00021/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning](https://mlsanity.com/paper/2312.10160/) (2023)
- [ICDAR 2019 Competition on Scene Text Visual Question Answering](https://mlsanity.com/paper/1907.00490/) (2019)
- [ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules](https://mlsanity.com/paper/2304.02173/) (2023)
- [Chart-Text: A Fully Automated Chart Image Descriptor](https://mlsanity.com/paper/1812.10636/) (2018)

## Built on by (most cited first, 7 of 7)

- [Protecting multimodal large language models against misleading visualizations](https://mlsanity.com/paper/2502.20503/) (2025)
- [Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trends](https://mlsanity.com/paper/2410.13883/) (2024)
- [Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding](https://mlsanity.com/paper/2504.09764/) (2025)
- [Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approach](https://mlsanity.com/paper/2506.06175/) (2025)
- [Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models](https://mlsanity.com/paper/2604.03157/) (2026)
- [Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation](https://mlsanity.com/paper/2604.24987/) (2026)
- [CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning](https://mlsanity.com/paper/2608.02833/) (2026)

## Abstract (arXiv)

Recently, interpreting complex charts with logical reasoning has emerged as challenges due to the development of vision-language models. A prior state-of-the-art (SOTA) model has presented an end-to-end method that leverages the vision-language model to convert charts into table format utilizing Large Language Model (LLM) for reasoning. However, unlike natural images, charts contain a mix of essential and irrelevant information required for chart reasoning, and we discover that this characteristic can lower the performance of chart-to-table extraction. In this paper, we introduce SIMPLOT, a method designed to extract only the elements necessary for chart reasoning. The proposed method involves two steps: 1) training to mimic a simple plot that contains only the essential information from a complex chart for table extraction, followed by 2) performing reasoning based on the table. Our model enables accurate chart reasoning without the need for additional annotations or datasets, and its effectiveness is demonstrated through various experiments. Furthermore, we propose a novel prompt mimicking how human interpret charts for more accurate reasoning. Our source code is available at https://github.com/sangwu99/Simplot.