# LayerPeeler: Autoregressive Peeling for Layer-wise Image Vectorization

arXiv: https://arxiv.org/abs/2505.23740 · HTML: https://mlsanity.com/paper/2505.23740/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 126 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [Taming Rectified Flow for Inversion and Editing](https://mlsanity.com/paper/2411.04746/) (2024)
- [Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models](https://mlsanity.com/paper/2305.16807/) (2023)
- [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision](https://mlsanity.com/paper/2411.07199/) (2024)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [Stable Flow: Vital Layers for Training-Free Image Editing](https://mlsanity.com/paper/2411.14430/) (2024)
- [StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing](https://mlsanity.com/paper/2303.15649/) (2023)
- [Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models](https://mlsanity.com/paper/2402.12058/) (2024)
- [Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models](https://mlsanity.com/paper/2411.07232/) (2024)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing](https://mlsanity.com/paper/2410.18756/) (2024)
- [Text-to-Vector Generation with Neural Path Representation](https://mlsanity.com/paper/2405.10317/) (2024)
- [PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data](https://mlsanity.com/paper/2502.14397/) (2025)
- [ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis](https://mlsanity.com/paper/2106.04912/) (2021)

## Built on by (most cited first, 8 of 8)

- [Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback](https://mlsanity.com/paper/2604.20730/) (2026)
- [LottieGPT: Tokenizing Vector Animation for Autoregressive Generation](https://mlsanity.com/paper/2604.11792/) (2026)
- [AmodalSVG: Amodal Image Vectorization via Semantic Layer Peeling](https://mlsanity.com/paper/2604.10940/) (2026)
- [Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure](https://mlsanity.com/paper/2512.14336/) (2025)
- [Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition](https://mlsanity.com/paper/2511.17454/) (2025)
- [Workflow-Aware Structured Layer Decomposition for Illustration Production](https://mlsanity.com/paper/2603.14925/) (2026)
- [SemLayer: Semantic-aware Generative Segmentation and Layer Construction for Abstract Icons](https://mlsanity.com/paper/2603.24039/) (2026)
- [Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing](https://mlsanity.com/paper/2609.14657/) (2026)

## Abstract (arXiv)

Image vectorization is a powerful technique that converts raster images into vector graphics, enabling enhanced flexibility and interactivity. However, popular image vectorization tools struggle with occluded regions, producing incomplete or fragmented shapes that hinder editability. While recent advancements have explored optimization-based and learning-based layer-wise image vectorization, these methods face limitations in vectorization quality and flexibility. In this paper, we introduce LayerPeeler, a novel layer-wise image vectorization approach that addresses these challenges through a progressive simplification paradigm. The key to LayerPeeler's success lies in its autoregressive peeling strategy: by identifying and removing the topmost non-occluded layers while recovering underlying content, we generate vector graphics with complete paths and coherent layer structures. Our method leverages vision-language models to construct a layer graph that captures occlusion relationships among elements, enabling precise detection and description for non-occluded layers. These descriptive captions are used as editing instructions for a finetuned image diffusion model to remove the identified layers. To ensure accurate removal, we employ localized attention control that precisely guides the model to target regions while faithfully preserving the surrounding content. To support this, we contribute a large-scale dataset specifically designed for layer peeling tasks. Extensive quantitative and qualitative experiments demonstrate that LayerPeeler significantly outperforms existing techniques, producing vectorization results with superior path semantics, geometric regularity, and visual fidelity.