# LLM Code Customization with Visual Results: A Benchmark on TikZ

arXiv: https://arxiv.org/abs/2505.04670 · HTML: https://mlsanity.com/paper/2505.04670/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 28 of 28)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers](https://mlsanity.com/paper/2304.14400/) (2023)
- [Visual Programming for Text-to-Image Generation and Evaluation](https://mlsanity.com/paper/2305.15328/) (2023)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation](https://mlsanity.com/paper/2412.18150/) (2024)
- [Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts](https://mlsanity.com/paper/2308.03921/) (2023)
- [DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning](https://mlsanity.com/paper/2310.12128/) (2023)
- [PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback](https://mlsanity.com/paper/2502.00988/) (2025)
- [WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://mlsanity.com/paper/2502.18356/) (2025)
- [WebApp1K: A Practical Code-Generation Benchmark for Web App Development](https://mlsanity.com/paper/2408.00019/) (2024)
- [An Empirical Study on Low-Code Programming using Traditional vs Large Language Model Support](https://mlsanity.com/paper/2402.01156/) (2024)
- [RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale](https://mlsanity.com/paper/2406.16801/) (2024)
- [PECC: Problem Extraction and Coding Challenges](https://mlsanity.com/paper/2404.18766/) (2024)

## Built on by (most cited first, 5 of 5)

- [EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolution](https://mlsanity.com/paper/2604.09568/) (2026)
- [AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation](https://mlsanity.com/paper/2601.04540/) (2026)
- [Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ](https://mlsanity.com/paper/2608.13441/) (2026)
- [Imperfect Visual Verification for Code Edition : A Case Study on TikZ](https://mlsanity.com/paper/2606.15693/) (2026)
- [Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry](https://mlsanity.com/paper/2608.18111/) (2026)

## Abstract (arXiv)

With the rise of AI-based code generation, customizing existing code out of natural language instructions to modify visual results -such as figures or images -has become possible, promising to reduce the need for deep programming expertise. However, even experienced developers can struggle with this task, as it requires identifying relevant code regions (feature location), generating valid code variants, and ensuring the modifications reliably align with user intent. In this paper, we introduce vTikZ, the first benchmark designed to evaluate the ability of Large Language Models (LLMs) to customize code while preserving coherent visual outcomes. Our benchmark consists of carefully curated vTikZ editing scenarios, parameterized ground truths, and a reviewing tool that leverages visual feedback to assess correctness. Empirical evaluation with stateof-the-art LLMs shows that existing solutions struggle to reliably modify code in alignment with visual intent, highlighting a gap in current AI-assisted code editing approaches. We argue that vTikZ opens new research directions for integrating LLMs with visual feedback mechanisms to improve code customization tasks in various domains beyond TikZ, including image processing, art creation, Web design, and 3D modeling.