# ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation

arXiv: https://arxiv.org/abs/2507.19492 · HTML: https://mlsanity.com/paper/2507.19492/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning](https://mlsanity.com/paper/2311.10774/) (2023)
- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation](https://mlsanity.com/paper/2501.06598/) (2025)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversion](https://mlsanity.com/paper/2501.17887/) (2025)
- [Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation](https://mlsanity.com/paper/2502.14846/) (2025)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems](https://mlsanity.com/paper/2410.14179/) (2024)
- [mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning](https://mlsanity.com/paper/2404.01548/) (2024)

## Built on by (most cited first, 8 of 8)

- [ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch](https://mlsanity.com/paper/2601.13606/) (2026)
- [ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding](https://mlsanity.com/paper/2603.27064/) (2026)
- [Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders](https://mlsanity.com/paper/2603.06569/) (2026)
- [MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction](https://mlsanity.com/paper/2604.01600/) (2026)
- [Infinity-Parser2 Technical Report](https://mlsanity.com/paper/2607.07836/) (2026)
- [CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval](https://mlsanity.com/paper/2604.15663/) (2026)
- [When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering](https://mlsanity.com/paper/2608.05670/) (2026)
- [ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications](https://mlsanity.com/paper/2610.08106/) (2026)

## Abstract (arXiv)

Chart-to-code reconstruction -- the task of recovering executable plotting scripts from chart images -- provides important insights into a model's ability to ground data visualizations in precise, machine-readable form. Yet many existing multimodal benchmarks largely focus primarily on answering questions about charts or summarizing them. To bridge this gap, we present ChartGen, a fully-automated pipeline for code-guided synthetic chart generation. Starting from seed chart images, ChartGen (i) prompts a vision-language model (VLM) to reconstruct each image into a python script, and (ii) iteratively augments that script with a code-oriented large language model (LLM). Using ChartGen, we create 222.5K unique chart-image code pairs from 13K seed chart images, and present an open-source synthetic chart dataset covering 27 chart types, 11 plotting libraries, and multiple data modalities (image, code, text, CSV, DocTags). From this corpus, we curate a held-out chart-to-code evaluation subset of 4.3K chart image-code pairs, and evaluate six open-weight VLMs (3B - 26B parameters), highlighting substantial room for progress. We release the pipeline, prompts, and the dataset to help accelerate efforts towards robust chart understanding and vision-conditioned code generation: https://github.com/SD122025/ChartGen/