mlsanityData to 14 Oct 2026

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision

arXiv 2410.05991 ↗First posted 8 Oct 2024cs.CV cs.AI cs.GRcited by 3

Not on the map. Why?

Abstract (arXiv)

Scalable Vector Graphics (SVG) is a popular format on the web and in the design industry. However, despite the great strides made in generative modeling, SVG has remained underexplored due to the discrete and complex nature of such data. We introduce GRIMOIRE, a text-guided SVG generative model that is comprised of two modules: A Visual Shape Quantizer (VSQ) learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an Auto-Regressive Transformer (ART) models the joint probability distribution over shape tokens, positions and textual descriptions, allowing us to generate vector graphics from natural language. Unlike existing models that require direct supervision from SVG data, GRIMOIRE learns shape image patches using only raster image supervision which opens up vector generative modeling to significantly more data. We demonstrate the effectiveness of our method by fitting GRIMOIRE for closed filled shapes on the MNIST and for outline strokes on icon and font data, surpassing previous image-supervised methods in generative quality and vector-supervised approach in flexibility.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

28 of 40 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  11. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  12. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
16 more
  1. 2017A Neural Representation of Sketch Drawingscited by 313
  2. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 187
  3. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 159
  4. 2014Neural Variational Inference and Learning in Belief Networkscited by 117
  5. 2021Im2Vec: Synthesizing Vector Graphics without Vector Supervisioncited by 116
  6. 2022VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelscited by 112
  7. 2019A Learned Representation for Scalable Vector Graphicscited by 91
  8. 2023IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformerscited by 75
  9. 2018Synthesizing Programs for Images using Reinforced Adversarial Learningcited by 72
  10. 2023Word-As-Image for Semantic Typographycited by 63
  11. 2019FIGR: Few-shot Image Generation with Reptilecited by 56
  12. 2022FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Contextcited by 49
  13. 2019Neural Painters: A learned differentiable constraint for generating brushstroke paintingscited by 28
  14. 2024StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesiscited by 26
  15. 2016Semi-Supervised Learning with Context-Conditional Generative Adversarial Networkscited by 19
  16. 2022SVG Vector Font Generation for Chinese Characters with Transformercited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Scalable Vector Graphics Latent Spacecited by 1
  2. 2026CANVAS: Consistency-Aware Navigation via Visual Adaptive Sampling for Long-Context Text-to-SVG Generationcited by 0
  3. 2026SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models
  2. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Model
  3. 2025SVGThinker: Instruction-Aligned and Reasoning-Driven Text-to-SVG Generation
  4. 2025VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics
  5. 2023SVGDreamer: Text Guided SVG Generation with Diffusion Model
  6. 2023Beyond Pixels: Exploring Human-Readable SVG Generation for Simple Images with Vision Language Models
  7. 2026IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework
  8. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Text