Imperfect Visual Verification for Code Edition : A Case Study on TikZ
Not on the map. Why?
Abstract (arXiv)
LLMs have significantly advanced code generation, enabling the synthesis of functional programs. While recent systems achieve strong performance on many coding benchmarks, tasks involving programs such as TikZ that generate visual artifacts remain challenging, in particular on visual code customization. Unlike generation from scratch, customization requires localized, semantics-preserving edits: the model must locate relevant code, modify it according to the instruction, and preserve the remaining structure and rendering. Approaches based on post-hoc iterative refinement/correction where a verifier provides feedback to guide corrections, have shown promise. However, in the case of programs with a visual outcome such as in TikZ, where correctness is harder or likely impossible to formalize and evaluate automatically, deterministic verifiers do not exist. Hence, developers can only rely on imperfect verifiers. In this paper, we conduct an empirical study to answer:to what extent can iterative refinement remain effective when the verifier itself is unreliable?} We use TikZ as a focused case study that isolates the core difficulties of the problem (weak code structure, fine-grained visual semantics, and difficult feature localization) in a controlled and challenging setting. We define visual code customization as an iterative editing problem with an imperfect oracle, and introduce a framework for analyzing such iterative refinements. We conduct a large-scale study and evaluate multiple LLM-based and tool-augmented visual verifiers within iterative refinement pipelines, and perform extensive manual annotation of refinement trajectories to assess verifier behavior and feedback quality. Our findings show that even imperfect verifiers can determine with moderate accuracy whether visual instructions are applied to code, achieving F1-scores up to 0.815. Feedback improves iterative refinement, especially for weaker models, adding 11--20 perfect customizations for Qwen3-vl-30b-a3b-Instruct, while stronger models like Gemini-3 gain fewer improvements (+5) but benefit more from accurate verification that prevents premature acceptance. Feedback is effective only when it precisely identifies image issues, provides actionable guidance, addresses all relevant problems, and remains grounded in the original instruction.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 9 Apr 2026.
Builds on
31 of 51 references.
References · 31
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Teaching Large Language Models to Self-Debugcited by 963
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
- 2022A Systematic Evaluation of Large Language Models of Codecited by 360
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
- 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
- 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
- 2023Shepherd: A Critic for Language Model Generationcited by 96
- 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
- 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
19 more
- 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
- 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
- 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 33
- 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
- 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 23
- 2026PaperBanana: Automating Academic Illustration for AI Scientistscited by 22
- 2024Prompt Chaining or Stepwise Prompt? Refinement in Text Summarizationcited by 16
- 2025VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representationcited by 15
- 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
- 2024CriticEval: Evaluating Large Language Model as Criticcited by 13
- 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
- 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiquescited by 12
- 2025VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generationcited by 8
- 2025Improved Iterative Refinement for Chart-to-Code Generation via Structured Instructioncited by 8
- 2024Multimodal Auto Validation For Self-Refinement in Web Agentscited by 5
- 2025LLM Code Customization with Visual Results: A Benchmark on TikZcited by 5
- 2025SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinementcited by 5
- 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
- 2025IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignmentcited by 1
Built on by
Not cited yet.