mlsanityData to 7 Oct 2026

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

arXiv 2607.27699 ↗First posted 30 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-based approaches rely on single-pass open-loop inference, where the model receives visual input only once and must generate thousands of SVG code tokens without intermediate verification. This paradigm inevitably leads to geometric drift, error accumulation, and visual hallucination on complex images. RefineSVG overcomes this limitation by invoking an external rendering engine after an initial SVG generation pass to compare the rendered output against the target image. The comparison yields a multi-dimensional visual residual map (Diff-Map) that is fed back to the model as a ReAct-style correction signal, driving a targeted correction step. To support this render-observe-correct interaction, we further introduce an SVG-oriented semantic vocabulary that compresses token sequences by over 52%. A progressive training pipeline spanning supervised fine-tuning, rejection-sampling cold-start data construction, and end-to-end agentic reinforcement learning aligns the model with closed-loop visual correction. Extensive experiments show that RefineSVG consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code efficiency.Code is available at https://github.com/liuxiaobo66/RefineSVG.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 41 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

21 of 41 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023Teaching Large Language Models to Self-Debugcited by 963
9 more
  1. 2023The False Promise of Imitating Proprietary LLMscited by 254
  2. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  3. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 158
  4. 2021Im2Vec: Synthesizing Vector Graphics without Vector Supervisioncited by 114
  5. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  6. 2023SVGDreamer: Text Guided SVG Generation with Diffusion Modelcited by 75
  7. 2023IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformerscited by 73
  8. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  9. 2025Rendering-Aware Reinforcement Learning for Vector Graphics Generationcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback
  2. 2026Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs
  3. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
  4. 2026Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning
  5. 2026IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework
  6. 2026Reinforcement Learning from Intermediate Renders for Image-to-Code Generation
  7. 2025SVGThinker: Instruction-Aligned and Reasoning-Driven Text-to-SVG Generation
  8. 2026GeoSVG-RL: Geometry-Aware Reinforcement Learning for Layout-Constrained Text-to-SVG Diagram Generation