mlsanityData to 7 Oct 2026

Character-Aware Models Improve Visual Text Rendering

arXiv 2212.10562 ↗First posted 20 Dec 2022cs.CL cs.CVcited by 90

Not on the map. Why?

Cited by 39 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Current image generation models struggle to reliably produce well-formed visual text. In this paper, we investigate a key contributing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word's visual makeup as a series of glyphs. To quantify this effect, we conduct a series of experiments comparing character-aware vs. character-blind text encoders. In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell). Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark). Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 23 of its 182 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

23 of 182 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
11 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  4. 2019How multilingual is Multilingual BERT?cited by 642
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  7. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  8. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  10. 2022What do tokens know about their characters and how do they know it?cited by 36
  11. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 33; 2025: 19; 2026: 15.

90 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  5. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  6. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  7. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  8. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  9. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  10. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 127
  11. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  12. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
38 more
  1. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  2. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 58
  3. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  4. 2023Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Modelscited by 50
  5. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  6. 2024Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Renderingcited by 47
  7. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  8. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  9. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  10. 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 44
  11. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  12. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  13. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  14. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  15. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generationcited by 34
  16. 2024PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generationcited by 34
  17. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  18. 2023Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedbackcited by 31
  19. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30
  20. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 28
  21. 2025TextAtlas5M: A Large-scale Dataset for Dense Text Image Generationcited by 25
  22. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  23. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  24. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  25. 2024Training LLMs over Neurally Compressed Textcited by 16
  26. 2024First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blendingcited by 16
  27. 2025WordCon: Word-level Typography Control in Scene Text Renderingcited by 16
  28. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  29. 2024Layout Agnostic Scene Text Image Synthesis with Diffusion Modelscited by 14
  30. 2024CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encodercited by 14
  31. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generationcited by 13
  32. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  33. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  34. 2023Towards Diverse and Consistent Typography Generationcited by 11
  35. 2024AMO Sampler: Enhancing Text Rendering with Overshootingcited by 11
  36. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  37. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  38. 2024Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Modelscited by 9

50 most cited of 90 All 90 (JSON)