# Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approach

arXiv: https://arxiv.org/abs/2506.06175 · HTML: https://mlsanity.com/paper/2506.06175/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 23 of 23)

- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models](https://mlsanity.com/paper/2312.08962/) (2023)
- [Small LLMs Are Weak Tool Learners: A Multi-LLM Agent](https://mlsanity.com/paper/2401.07324/) (2024)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [Visualization Generation with Large Language Models: An Evaluation](https://mlsanity.com/paper/2401.11255/) (2024)
- [TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering](https://mlsanity.com/paper/2404.01476/) (2024)
- [Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector](https://mlsanity.com/paper/2406.11277/) (2024)
- [Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness](https://mlsanity.com/paper/2407.11229/) (2024)
- [Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback](https://mlsanity.com/paper/2410.04064/) (2024)
- [ChartReformer: Natural Language-Driven Chart Image Editing](https://mlsanity.com/paper/2403.00209/) (2024)
- [VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction](https://mlsanity.com/paper/2310.09611/) (2023)
- [Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations](https://mlsanity.com/paper/2402.02167/) (2024)
- [SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials](https://mlsanity.com/paper/2405.00021/) (2024)
- [C2: Scalable Auto-Feedback for LLM-based Chart Generation](https://mlsanity.com/paper/2410.18652/) (2024)
- [V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and Suggestions](https://mlsanity.com/paper/2406.15259/) (2024)
- [PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents](https://mlsanity.com/paper/2501.11233/) (2025)
- [Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations](https://mlsanity.com/paper/2409.18764/) (2024)

## Built on by (most cited first, 1 of 1)

- [ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement](https://mlsanity.com/paper/2512.13303/) (2025)

## Abstract (arXiv)

Large language models can translate natural-language chart descriptions into runnable code, yet approximately 15% of the generated scripts still fail to execute, even after supervised fine-tuning and reinforcement learning. We investigate whether this persistent error rate stems from model limitations or from reliance on a single-prompt design. To explore this, we propose a lightweight multi-agent pipeline that separates drafting, execution, repair, and judgment, using only an off-the-shelf GPT-4o-mini model. On the textscText2Chart31 benchmark, our system reduces execution errors to 4.5% within three repair iterations, outperforming the strongest fine-tuned baseline by nearly 5 percentage points while requiring significantly less compute. Similar performance is observed on the textscChartX benchmark, with an error rate of 4.6%, demonstrating strong generalization. Under current benchmarks, execution success appears largely solved. However, manual review reveals that 6 out of 100 sampled charts contain hallucinations, and an LLM-based accessibility audit shows that only 33.3% (textscText2Chart31) and 7.2% (textscChartX) of generated charts satisfy basic colorblindness guidelines. These findings suggest that future work should shift focus from execution reliability toward improving chart aesthetics, semantic fidelity, and accessibility.