mlsanityData to 7 Oct 2026

TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering

arXiv 2311.16465 ↗First posted 28 Nov 2023cs.CVcited by 136

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The diffusion model has been proven a powerful generative model in recent years, yet remains a challenge in generating visual text. Several methods alleviated this issue by incorporating explicit text position and content as guidance on where and what text to render. However, these methods still suffer from several drawbacks, such as limited flexibility and automation, constrained capability of layout prediction, and restricted style diversity. In this paper, we present TextDiffuser-2, aiming to unleash the power of language models for text rendering. Firstly, we fine-tune a large language model for layout planning. The large language model is capable of automatically generating keywords for text rendering and also supports layout modification through chatting. Secondly, we utilize the language model within the diffusion model to encode the position and texts at the line level. Unlike previous methods that employed tight character-level guidance, this approach generates more diverse text images. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V, validating TextDiffuser-2's capacity to achieve a more rational text layout and generation with enhanced diversity. The code and model will be available at urlhttps://aka.ms/textdiffuser-2.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 59 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

52 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023GlyphControl: Glyph Conditional Control for Visual Text Generation29 May
  2. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  3. 2023TextDiffuser: Diffusion Models as Text Painters18 May
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
32 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  4. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  7. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  8. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  10. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  11. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  12. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
  13. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  14. 2022A Unified Sequence Interface for Vision Taskscited by 152
  15. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  16. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  17. 2016Discrete Variational Autoencoderscited by 124
  18. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 122
  19. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  20. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  21. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  22. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  23. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  24. 2021Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Studycited by 52
  25. 2019READ: Recursive Autoencoders for Document Layout Generationcited by 51
  26. 2023LayoutPrompter: Awaken the Design Ability of Large Language Modelscited by 47
  27. 2018Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentationcited by 42
  28. 2022Discovering the Hidden Vocabulary of DALLE-2cited by 42
  29. 2021MOST: A Multi-Oriented Scene Text Detector with Localization Refinementcited by 33
  30. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehensioncited by 27
  31. 2023Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligningcited by 24
  32. 2023Diffusion-based Document Layout Generationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 49; 2026: 47.

136 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  3. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  4. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  5. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  6. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  7. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  8. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  9. 2024Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Renderingcited by 47
  10. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  11. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  12. 2024PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLMcited by 39
38 more
  1. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  2. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  3. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  4. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  7. 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 29
  8. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
  9. 2025TextAtlas5M: A Large-scale Dataset for Dense Text Image Generationcited by 25
  10. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  11. 2024Visual Text Generation in the Wildcited by 22
  12. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  13. 2025Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generationcited by 22
  14. 2024Revisiting Tampered Scene Text Detection in the Era of Generative AIcited by 21
  15. 2024Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editingcited by 18
  16. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  17. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  18. 2025WordCon: Word-level Typography Control in Scene Text Renderingcited by 16
  19. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15
  20. 2025DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Designcited by 15
  21. 2024Can GPTs Evaluate Graphic Design Based on Design Principles?cited by 14
  22. 2024CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encodercited by 14
  23. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 14
  24. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generationcited by 13
  25. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  26. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  27. 2024JoyType: A Robust Design for Multilingual Visual Text Creationcited by 12
  28. 2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesiscited by 12
  29. 2024AMO Sampler: Enhancing Text Rendering with Overshootingcited by 11
  30. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  31. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  32. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  33. 2026JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generationcited by 10
  34. 2024Kinetic Typography Diffusion Modelcited by 9
  35. 2024How Control Information Influences Multilingual Text Image Generation and Editing?cited by 9
  36. 2024Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generationcited by 9
  37. 2024A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generationcited by 9
  38. 2025PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agentscited by 9

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  2. 2024Layout Agnostic Scene Text Image Synthesis with Diffusion Models
  3. 2024CustomText: Customized Textual Image Generation using Diffusion Models
  4. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation
  5. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
  6. 2024Conditional Text-to-Image Generation with Reference Guidance
  7. 2026UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
  8. 2026Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement