# Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

arXiv: https://arxiv.org/abs/2312.12232 · HTML: https://mlsanity.com/paper/2312.12232/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](https://mlsanity.com/paper/2305.10474/) (2023)
- [SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction](https://mlsanity.com/paper/2310.20700/) (2023)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [Paint by Word](https://mlsanity.com/paper/2103.10951/) (2021)
- [ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)](https://mlsanity.com/paper/1708.09585/) (2017)
- [Editing Text in the Wild](https://mlsanity.com/paper/1908.03047/) (2019)
- [DG-Font: Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2104.03064/) (2021)
- [Spatial Fusion GAN for Image Synthesis](https://mlsanity.com/paper/1812.05840/) (2018)
- [SwapText: Image Based Texts Transfer in Scenes](https://mlsanity.com/paper/2003.08152/) (2020)
- [TextStyleBrush: Transfer of Text Aesthetics from a Single Example](https://mlsanity.com/paper/2106.08385/) (2021)
- [UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World](https://mlsanity.com/paper/2003.10608/) (2020)

## Built on by (most cited first, 30 of 41)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [AnyText2: Visual Text Generation and Editing With Customizable Attributes](https://mlsanity.com/paper/2411.15245/) (2024)
- [FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing](https://mlsanity.com/paper/2505.03329/) (2025)
- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models](https://mlsanity.com/paper/2407.02252/) (2024)
- [EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering](https://mlsanity.com/paper/2505.24417/) (2025)
- [Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generation](https://mlsanity.com/paper/2503.23461/) (2025)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [DiffusionPen: Towards Controlling the Style of Handwritten Text Generation](https://mlsanity.com/paper/2409.06065/) (2024)
- [First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending](https://mlsanity.com/paper/2410.10168/) (2024)
- [TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering](https://mlsanity.com/paper/2602.20903/) (2026)
- [CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encoder](https://mlsanity.com/paper/2412.17225/) (2024)
- [Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation](https://mlsanity.com/paper/2403.16422/) (2024)
- [TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis](https://mlsanity.com/paper/2505.17778/) (2025)
- [JoyType: A Robust Design for Multilingual Visual Text Creation](https://mlsanity.com/paper/2409.17524/) (2024)
- [ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotations](https://mlsanity.com/paper/2502.10999/) (2025)
- [T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models](https://mlsanity.com/paper/2505.04946/) (2025)
- [Rethinking Layered Graphic Design Generation with a Top-Down Approach](https://mlsanity.com/paper/2507.05601/) (2025)
- [PosterVerse: A Full-Workflow Framework for Commercial-Grade Poster Generation with HTML-Based Scalable Typography](https://mlsanity.com/paper/2601.03993/) (2026)
- [Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generation](https://mlsanity.com/paper/2407.14774/) (2024)
- [UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis](https://mlsanity.com/paper/2507.00992/) (2025)
- [Type-R: Automatically Retouching Typos for Text-to-Image Generation](https://mlsanity.com/paper/2411.18159/) (2024)
- [Exploring In-Image Machine Translation with Real-World Background](https://mlsanity.com/paper/2505.15282/) (2025)
- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)
- [UTDesign: A Unified Framework for Stylized Text Editing and Generation in Graphic Design Images](https://mlsanity.com/paper/2512.20479/) (2025)
- [Typographic Text Generation with Off-the-Shelf Diffusion Model](https://mlsanity.com/paper/2402.14314/) (2024)
- [Precise Parameter Localization for Textual Generation in Diffusion Models](https://mlsanity.com/paper/2502.09935/) (2025)
- [TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark](https://mlsanity.com/paper/2503.13730/) (2025)
- [TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards](https://mlsanity.com/paper/2605.19320/) (2026)
- [TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation](https://mlsanity.com/paper/2404.11824/) (2024)

## Abstract (arXiv)

Recently, diffusion-based image generation methods are credited for their remarkable text-to-image generation capabilities, while still facing challenges in accurately generating multilingual scene text images. To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language. Our model outputs a photo-realistic image given a text of any language along with a textual description of a scene. The model leverages rendered sketch images as priors, thus arousing the potential multilingual-generation ability of the pre-trained Stable Diffusion. Based on the observation from the influence of the cross-attention map on object placement in generated images, we propose a localized attention constraint into the cross-attention layer to address the unreasonable positioning problem of scene text. Additionally, we introduce contrastive image-level prompts to further refine the position of the textual region and achieve more accurate scene text generation. Experiments demonstrate that our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.