mlsanityData to 7 Oct 2026

TextDiffuser: Diffusion Models as Text Painters

arXiv 2305.10855 ↗First posted 18 May 2023cs.CVcited by 223

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at urlhttps://aka.ms/textdiffuser.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 82 of its 107 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

82 of 107 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoders12 Apr
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2022Diff-Font: Diffusion Model for Robust One-Shot Font Generation12 Dec
  4. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts27 Oct
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  11. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  12. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  13. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  16. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  17. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  18. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  19. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  20. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  21. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  22. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  23. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  24. 2019Pluralistic Image Completioncited by 162
  25. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  26. 2019Shape Robust Text Detection with Progressive Scale Expansion Networkcited by 147
  27. 2021High-Fidelity Pluralistic Image Completion with Transformerscited by 134
  28. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  29. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  30. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  31. 2019StructureFlow: Image Inpainting via Structure-aware Appearance Flowcited by 118
  32. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  33. 2020Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizationscited by 102
  34. 2019Learning Pyramid-Context Encoder Network for High-Quality Image Inpaintingcited by 94
  35. 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
  36. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  37. 2019Editing Text in the Wildcited by 81
  38. 2021Diverse Image Inpainting with Bidirectional and Autoregressive Transformerscited by 80

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 65; 2025: 65; 2026: 80.

223 papers.

Sort:

By area:

  1. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  2. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Modelscited by 138
  3. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  4. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  5. 2023AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusioncited by 98
  6. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  7. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  9. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  10. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  11. 2023SVGDreamer: Text Guided SVG Generation with Diffusion Modelcited by 75
  12. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
38 more
  1. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 58
  2. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  3. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  4. 2024Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Renderingcited by 47
  5. 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 44
  6. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  7. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Modelcited by 41
  8. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  9. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  10. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  11. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  12. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  13. 2025AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Modelscited by 33
  14. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  15. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  16. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 28
  17. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
  18. 2025TextAtlas5M: A Large-scale Dataset for Dense Text Image Generationcited by 25
  19. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  20. 2024Visual Text Generation in the Wildcited by 22
  21. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  22. 2025Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generationcited by 22
  23. 2024Revisiting Tampered Scene Text Detection in the Era of Generative AIcited by 21
  24. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  25. 2023KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesiscited by 18
  26. 2024Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editingcited by 18
  27. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  28. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  29. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  30. 2024Bridging Different Language Models and Generative Vision Models for Text-to-Image Generationcited by 17
  31. 2024DreamText: High Fidelity Scene Text Synthesiscited by 17
  32. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  33. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  34. 2024DiffusionPen: Towards Controlling the Style of Handwritten Text Generationcited by 16
  35. 2024First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blendingcited by 16
  36. 2025WordCon: Word-level Typography Control in Scene Text Renderingcited by 16
  37. 2023Planning and Rendering: Towards Product Poster Generation with Diffusion Modelscited by 15
  38. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TextDiffuser-RL: Efficient and Robust Text Layout Optimization for High-Fidelity Text-to-Image Synthesis
  2. 2023Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models
  3. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation
  4. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  5. 2023DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
  6. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
  7. 2023Zero-shot spatial layout conditioning for text-to-image diffusion models
  8. 2024Contextualized Diffusion Models for Text-Guided Image and Video Generation