mlsanityData to 7 Oct 2026

GlyphControl: Glyph Conditional Control for Visual Text Generation

arXiv 2305.18259 ↗First posted 29 May 2023cs.CVcited by 127

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, there has been an increasing interest in developing diffusion-based text-to-image generative models capable of generating coherent and well-formed visual text. In this paper, we propose a novel and efficient approach called GlyphControl to address this task. Unlike existing methods that rely on character-aware text encoders like ByT5 and require retraining of text-to-image models, our approach leverages additional glyph conditional information to enhance the performance of the off-the-shelf Stable-Diffusion model in generating accurate visual text. By incorporating glyph instructions, users can customize the content, location, and size of the generated text according to their specific requirements. To facilitate further research in visual text generation, we construct a training benchmark dataset called LAION-Glyph. We evaluate the effectiveness of our approach by measuring OCR-based metrics, CLIP score, and FID of the generated visual text. Our empirical evaluations demonstrate that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR accuracy, CLIP score, and FID, highlighting the efficacy of our method.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 29 of its 35 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

29 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition25 May
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022Diff-Font: Diffusion Model for Robust One-Shot Font Generation12 Dec
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 20

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
8 more
  1. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  2. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  3. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  4. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  5. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  6. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  7. 2022SinFusion: Training Diffusion Models on a Single Image or Videocited by 58
  8. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 40; 2025: 45; 2026: 32.

127 papers.

Sort:

By area:

  1. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  2. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  3. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  4. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  5. 2023SVGDreamer: Text Guided SVG Generation with Diffusion Modelcited by 75
  6. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 58
  7. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  8. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  9. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  10. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  11. 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 44
  12. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
38 more
  1. 2025PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Renderingcited by 41
  2. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  3. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  4. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
  7. 2025TextAtlas5M: A Large-scale Dataset for Dense Text Image Generationcited by 25
  8. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  9. 2023Implicit Concept Removal of Diffusion Modelscited by 23
  10. 2024Visual Text Generation in the Wildcited by 22
  11. 2025Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generationcited by 22
  12. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  13. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  14. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  15. 2024First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blendingcited by 16
  16. 2025WordCon: Word-level Typography Control in Scene Text Renderingcited by 16
  17. 2023WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Modelscited by 15
  18. 2023Planning and Rendering: Towards Product Poster Generation with Diffusion Modelscited by 15
  19. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15
  20. 2024Layout Agnostic Scene Text Image Synthesis with Diffusion Modelscited by 14
  21. 2024CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encodercited by 14
  22. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 14
  23. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generationcited by 13
  24. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  25. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  26. 2024JoyType: A Robust Design for Multilingual Visual Text Creationcited by 12
  27. 2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesiscited by 12
  28. 2023Towards Diverse and Consistent Typography Generationcited by 11
  29. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  30. 2025GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editingcited by 10
  31. 2025PICD: Versatile Perceptual Image Compression with Diffusion Renderingcited by 10
  32. 2024Kinetic Typography Diffusion Modelcited by 9
  33. 2024How Control Information Influences Multilingual Text Image Generation and Editing?cited by 9
  34. 2024Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generationcited by 9
  35. 2024A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generationcited by 9
  36. 2025T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Modelscited by 9
  37. 2024Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processingcited by 8
  38. 2024TextSSR: Diffusion-based Data Synthesis for Scene Text Recognitioncited by 8

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GlyphDiffusion: Text Generation as Image Generation
  2. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generation
  3. 2025Glyph: Scaling Context Windows via Visual-Text Compression
  4. 2025RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation
  5. 2026Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering
  6. 2025GlyphWeaver: Unlocking Glyph Design Creativity with Uniform Glyph DSL and AI ↗
  7. 2026SteerVTE: Seamless Video Text Editing with Style and Glyph Control
  8. 2023Controllable Text-to-Image Generation with GPT-4