mlsanityData to 7 Oct 2026

Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering

arXiv 2403.09622 ↗First posted 14 Mar 2024cs.CVcited by 80

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual text rendering poses a fundamental challenge for contemporary text-to-image generation models, with the core problem lying in text encoder deficiencies. To achieve accurate text rendering, we identify two crucial requirements for text encoders: character awareness and alignment with glyphs. Our solution involves crafting a series of customized text encoder, Glyph-ByT5, by fine-tuning the character-aware ByT5 encoder using a meticulously curated paired glyph-text dataset. We present an effective method for integrating Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for design image generation. This significantly enhances text rendering accuracy, improving it from less than 20% to nearly 90% on our design image benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph rendering, achieving high spelling accuracy for tens to hundreds of characters with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with a small set of high-quality, photorealistic images featuring visual text, we showcase a substantial improvement in scene text rendering capabilities in open-domain real images. These compelling outcomes aim to encourage further exploration in designing customized text encoders for diverse and challenging tasks.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

27 of 33 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  12. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
15 more
  1. 2022Blended Latent Diffusioncited by 430
  2. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  3. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  4. 2023Teaching CLIP to Count to Tencited by 194
  5. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  6. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 127
  7. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  8. 2021Vision Transformer for Fast and Efficient Scene Text Recognitioncited by 81
  9. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 58
  10. 2020Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approachcited by 45
  11. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  12. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  13. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  14. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  15. 2023Customization Assistant for Text-to-image Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 38; 2026: 31.

80 papers.

Sort:

By area:

  1. 2025HunyuanVideo 1.5 Technical Reportcited by 148
  2. 2025LongCat-Image Technical Reportcited by 92
  3. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  4. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  5. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  6. 2024Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Renderingcited by 47
  7. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  8. 2025PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Renderingcited by 41
  9. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  10. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  11. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  12. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
38 more
  1. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  2. 2025Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generationcited by 22
  3. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  4. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  5. 2024DreamText: High Fidelity Scene Text Synthesiscited by 17
  6. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15
  7. 2025DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Designcited by 15
  8. 2025PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Modelscited by 14
  9. 2025X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillationcited by 11
  10. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  11. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  12. 2024How Control Information Influences Multilingual Text Image Generation and Editing?cited by 9
  13. 2024A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generationcited by 9
  14. 2025Rethinking Layered Graphic Design Generation with a Top-Down Approachcited by 9
  15. 2025UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesiscited by 8
  16. 2025STRICT: Stress Test of Rendering Images Containing Textcited by 7
  17. 2025IGD: Instructional Graphic Design with Multimodal Layer Generationcited by 6
  18. 2026WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editingcited by 6
  19. 2024Type-R: Automatically Retouching Typos for Text-to-Image Generationcited by 5
  20. 2025T-Stars-Poster: A Framework for Product-Centric Advertising Image Designcited by 5
  21. 2025TextEditBench: Evaluating Reasoning-aware Text Editing Beyond Renderingcited by 5
  22. 2026InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generationcited by 5
  23. 2024TextMaster: A Unified Framework for Realistic Text Editing via Glyph-Style Dual-Controlcited by 4
  24. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  25. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  26. 2026UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editingcited by 4
  27. 2026GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflowscited by 4
  28. 2026GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Renderingcited by 4
  29. 2026AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generationcited by 4
  30. 2026TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewardscited by 4
  31. 2025Autoregressive Styled Text Image Generation, but Make it Reliablecited by 3
  32. 2025Video Text Preservation with Synthetic Text-Rich Videoscited by 3
  33. 2025DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategycited by 3
  34. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 3
  35. 2026simpleposter: A simple baseline for product poster generationcited by 3
  36. 2026StyleTextGen: Style-Conditioned Multilingual Scene Text Generationcited by 3
  37. 2026SteerVTE: Seamless Video Text Editing with Style and Glyph Controlcited by 3
  38. 2024CustomText: Customized Textual Image Generation using Diffusion Modelscited by 2

50 most cited of 80 All 80 (JSON)