mlsanityData to 7 Oct 2026

Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

arXiv 2312.12232 ↗First posted 19 Dec 2023cs.CVcited by 41

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, diffusion-based image generation methods are credited for their remarkable text-to-image generation capabilities, while still facing challenges in accurately generating multilingual scene text images. To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language. Our model outputs a photo-realistic image given a text of any language along with a textual description of a scene. The model leverages rendered sketch images as priors, thus arousing the potential multilingual-generation ability of the pre-trained Stable Diffusion. Based on the observation from the influence of the cross-attention map on object placement in generated images, we propose a localized attention constraint into the cross-attention layer to address the unreasonable positioning problem of scene text. Additionally, we introduce contrastive image-level prompts to further refine the position of the textual region and achieve more accurate scene text generation. Experiments demonstrate that our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 35 of its 45 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

35 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023TextDiffuser: Diffusion Models as Text Painters18 May
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022Diff-Font: Diffusion Model for Robust One-Shot Font Generation12 Dec
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  11. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  12. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
13 more
  1. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  2. 2021Paint by Wordcited by 107
  3. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  4. 2019Editing Text in the Wildcited by 81
  5. 2021DG-Font: Deformable Generative Networks for Unsupervised Font Generationcited by 70
  6. 2018Spatial Fusion GAN for Image Synthesiscited by 68
  7. 2020SwapText: Image Based Texts Transfer in Scenescited by 49
  8. 2021TextStyleBrush: Transfer of Text Aesthetics from a Single Examplecited by 48
  9. 2020UnrealText: Synthesizing Realistic Scene Text Images from the Unreal Worldcited by 43
  10. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  11. 2023LEO: Generative Latent Image Animator for Human Video Synthesiscited by 40
  12. 2019SynthText3D: Synthesizing Scene Text Images from 3D Virtual Worldscited by 15
  13. 2023Weakly Supervised Scene Text Generation for Low-resource Languagescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 21; 2026: 5.

41 papers.

Sort:

By area:

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  2. 2024AnyText2: Visual Text Generation and Editing With Customizable Attributescited by 48
  3. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  4. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  5. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 28
  6. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  7. 2025Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generationcited by 22
  8. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  9. 2024DiffusionPen: Towards Controlling the Style of Handwritten Text Generationcited by 16
  10. 2024First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blendingcited by 16
  11. 2024CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encodercited by 14
  12. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 14
29 more
  1. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generationcited by 13
  2. 2024JoyType: A Robust Design for Multilingual Visual Text Creationcited by 12
  3. 2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesiscited by 12
  4. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  5. 2024Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generationcited by 9
  6. 2025T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Modelscited by 9
  7. 2025Rethinking Layered Graphic Design Generation with a Top-Down Approachcited by 9
  8. 2025UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesiscited by 8
  9. 2026PosterVerse: A Full-Workflow Framework for Commercial-Grade Poster Generation with HTML-Based Scalable Typographycited by 8
  10. 2025Exploring In-Image Machine Translation with Real-World Backgroundcited by 7
  11. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  12. 2025UTDesign: A Unified Framework for Stylized Text Editing and Generation in Graphic Design Imagescited by 6
  13. 2024Typographic Text Generation with Off-the-Shelf Diffusion Modelcited by 5
  14. 2024Type-R: Automatically Retouching Typos for Text-to-Image Generationcited by 5
  15. 2025Precise Parameter Localization for Textual Generation in Diffusion Modelscited by 5
  16. 2025TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmarkcited by 4
  17. 2026TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewardscited by 4
  18. 2024TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generationcited by 3
  19. 2025Calligrapher: Freestyle Text Image Customizationcited by 3
  20. 2024AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Statuscited by 2
  21. 2025Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generationcited by 2
  22. 2025From Fragment to One Piece: A Survey on AI-Driven Graphic Designcited by 2
  23. 2024LTOS: Layout-controllable Text-Object Synthesis via Adaptive Cross-attention Fusionscited by 1
  24. 2025HDGlyph: A Hierarchical Disentangled Glyph-Based Framework for Long-Tail Text Rendering in Diffusion Modelscited by 1
  25. 2026POCA: Pareto-Optimal Curriculum Alignment for Visual Text Generationcited by 1
  26. 2024TextDestroyer: A Training- and Annotation-Free Diffusion Method for Destroying Anomal Text from Imagescited by 0
  27. 2025OrienText: Surface Oriented Textual Image Generationcited by 0
  28. 2025TripleFDS: Triple Feature Disentanglement and Synthesis for Scene Text Editingcited by 0
  29. 2026Planning and Rendering in Concert: DeepFusion of Autoregressive Layouts and Diffusion for Visual Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models
  2. 2024Layout Agnostic Scene Text Image Synthesis with Diffusion Models
  3. 2024Conditional Text-to-Image Generation with Reference Guidance
  4. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoders
  5. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
  6. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation
  7. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  8. 2023DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models