# OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

arXiv: https://arxiv.org/abs/2404.09987 · HTML: https://mlsanity.com/paper/2404.09987/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [FeTaQA: Free-form Table Question Answering](https://mlsanity.com/paper/2104.00369/) (2021)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning](https://mlsanity.com/paper/2311.10774/) (2023)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models](https://mlsanity.com/paper/2312.06109/) (2023)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning](https://mlsanity.com/paper/2307.09474/) (2023)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Merlin:Empowering Multimodal LLMs with Foresight Minds](https://mlsanity.com/paper/2312.00589/) (2023)

## Built on by (most cited first, 30 of 39)

- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)
- [DeepSeek-OCR 2: Visual Causal Flow](https://mlsanity.com/paper/2601.20552/) (2026)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning](https://mlsanity.com/paper/2404.16635/) (2024)
- [ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding](https://mlsanity.com/paper/2409.03277/) (2024)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [Slow Perception: Let's Perceive Geometric Figures Step-by-step](https://mlsanity.com/paper/2412.20631/) (2024)
- [PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training](https://mlsanity.com/paper/2606.03264/) (2026)
- [Effective Training Data Synthesis for Improving MLLM Chart Understanding](https://mlsanity.com/paper/2508.06492/) (2025)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [Perception in Reflection](https://mlsanity.com/paper/2504.07165/) (2025)
- [ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding](https://mlsanity.com/paper/2505.19076/) (2025)
- [StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond](https://mlsanity.com/paper/2405.21013/) (2024)
- [Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding](https://mlsanity.com/paper/2601.20430/) (2026)
- [SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More](https://mlsanity.com/paper/2601.05688/) (2026)
- [VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning](https://mlsanity.com/paper/2409.01667/) (2024)
- [Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts](https://mlsanity.com/paper/2503.04095/) (2025)
- [Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?](https://mlsanity.com/paper/2505.12766/) (2025)
- [ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning](https://mlsanity.com/paper/2512.00305/) (2025)
- [Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing](https://mlsanity.com/paper/2602.16455/) (2026)
- [ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoning](https://mlsanity.com/paper/2512.14040/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [Infinity-Parser2 Technical Report](https://mlsanity.com/paper/2607.07836/) (2026)
- [Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trends](https://mlsanity.com/paper/2410.13883/) (2024)
- [Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding](https://mlsanity.com/paper/2505.05446/) (2025)
- [In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding](https://mlsanity.com/paper/2507.14298/) (2025)

## Abstract (arXiv)

Chart parsing poses a significant challenge due to the diversity of styles, values, texts, and so forth. Even advanced large vision-language models (LVLMs) with billions of parameters struggle to handle such tasks satisfactorily. To address this, we propose OneChart: a reliable agent specifically devised for the structural extraction of chart information. Similar to popular LVLMs, OneChart incorporates an autoregressive main body. Uniquely, to enhance the reliability of the numerical parts of the output, we introduce an auxiliary token placed at the beginning of the total tokens along with an additional decoder. The numerically optimized (auxiliary) token allows subsequent tokens for chart parsing to capture enhanced numerical features through causal attention. Furthermore, with the aid of the auxiliary token, we have devised a self-evaluation mechanism that enables the model to gauge the reliability of its chart parsing results by providing confidence scores for the generated content. Compared to current state-of-the-art (SOTA) chart parsing models, e.g., DePlot, ChartVLM, ChartAst, OneChart significantly outperforms in Average Precision (AP) for chart structural extraction across multiple public benchmarks, despite enjoying only 0.2 billion parameters. Moreover, as a chart parsing agent, it also brings 10%+ accuracy gains for the popular LVLM (LLaVA-1.6) in the downstream ChartQA benchmark.