# MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Design

arXiv: https://arxiv.org/abs/2609.06116 · HTML: https://mlsanity.com/paper/2609.06116/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Paint by Word](https://mlsanity.com/paper/2103.10951/) (2021)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Sketch2code: Generating a website from a paper mockup](https://mlsanity.com/paper/1905.13750/) (2019)
- [Conditional Image Generation and Manipulation for User-Specified Content](https://mlsanity.com/paper/2005.04909/) (2020)
- [Exploring the Capability of LLMs in Performing Low-Level Visual Analytic Tasks on SVG Data Visualizations](https://mlsanity.com/paper/2404.19097/) (2024)
- [SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation](https://mlsanity.com/paper/2506.03139/) (2025)
- [Visually Descriptive Language Model for Vector Graphics Reasoning](https://mlsanity.com/paper/2404.06479/) (2024)
- [SVGEditBench V2: A Benchmark for Instruction-based SVG Editing](https://mlsanity.com/paper/2502.19453/) (2025)
- [Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding](https://mlsanity.com/paper/2306.06094/) (2023)
- [VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics](https://mlsanity.com/paper/2506.15903/) (2025)
- [Leveraging Large Language Models For Scalable Vector Graphics Processing: A Review](https://mlsanity.com/paper/2503.04983/) (2025)
- [Creating Language-driven Spatial Variations of Icon Images](https://mlsanity.com/paper/2405.19636/) (2024)

## Abstract (arXiv)

In the field of UI design, Scalable Vector Graphics (SVG) is widely used as a design medium. However, traditional SVG editing techniques have high entry barriers and require cumbersome manual iteration, while LLM-based editing solutions suffer from low accuracy and poor user controllability. To address these issues, we propose MM-SVGEdit, a multimodal-driven SVG editing approach that integrates traditional SVG editing and LLM-based methods. We introduce a two-stage strategy in which visual grounding is followed by modification. Both stages support two interaction modalities: natural language instructions and direct manipulation (mouse and keyboard). We trained and evaluated MM-SVGEdit on a self-constructed dataset of 14,476 question-answer pairs generated from UIs, covering 11 types of editing operations on both single and multiple UI targets. The results show that MM-SVGEdit improves SVG editing accuracy, efficiency, and user-perceived control while reducing token consumption and response time.