# Conditional Text Image Generation with Diffusion Models

arXiv: https://arxiv.org/abs/2306.10804 · HTML: https://mlsanity.com/paper/2306.10804/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 53 of its 77 references that are in the map.
- First posted on arXiv 19 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018)
- [Decoupled Attention Network for Text Recognition](https://mlsanity.com/paper/1912.10205/) (2019)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)

## Built on by (most cited first, 30 of 58)

- [FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning](https://mlsanity.com/paper/2312.12142/) (2023)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [Diffusion Model is a Good Pose Estimator from 3D RF-Vision](https://mlsanity.com/paper/2403.16198/) (2024)
- [Beyond Deepfake Images: Detecting AI-Generated Videos](https://mlsanity.com/paper/2404.15955/) (2024)
- [Visual Text Generation in the Wild](https://mlsanity.com/paper/2407.14138/) (2024)
- [One-Shot Diffusion Mimicker for Handwritten Text Generation](https://mlsanity.com/paper/2409.04004/) (2024)
- [VATr++: Choose Your Words Wisely for Handwritten Text Generation](https://mlsanity.com/paper/2402.10798/) (2024)
- [Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach](https://mlsanity.com/paper/2506.03979/) (2025)
- [DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Mode](https://mlsanity.com/paper/2503.12838/) (2025)
- [DiffusionPen: Towards Controlling the Style of Handwritten Text Generation](https://mlsanity.com/paper/2409.06065/) (2024)
- [Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation](https://mlsanity.com/paper/2410.17606/) (2024)
- [Zero-Shot Styled Text Image Generation, but Make It Autoregressive](https://mlsanity.com/paper/2503.17074/) (2025)
- [Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generation](https://mlsanity.com/paper/2508.03256/) (2025)
- [OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities](https://mlsanity.com/paper/2507.15085/) (2025)
- [Text Image Inpainting via Global Structure-Guided Diffusion Models](https://mlsanity.com/paper/2401.14832/) (2024)
- [TextSSR: Diffusion-based Data Synthesis for Scene Text Recognition](https://mlsanity.com/paper/2412.01137/) (2024)
- [E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data](https://mlsanity.com/paper/2404.08814/) (2024)
- [Rethinking HTG Evaluation: Bridging Generation and Recognition](https://mlsanity.com/paper/2409.02683/) (2024)
- [Diffusion Model Quantization: A Review](https://mlsanity.com/paper/2505.05215/) (2025)
- [Typographic Text Generation with Off-the-Shelf Diffusion Model](https://mlsanity.com/paper/2402.14314/) (2024)
- [Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models](https://mlsanity.com/paper/2409.00786/) (2024)
- [TRADES: Generating Realistic Market Simulations with Diffusion Models](https://mlsanity.com/paper/2502.07071/) (2025)
- [Growing Visual Generative Capacity for Pre-Trained MLLMs](https://mlsanity.com/paper/2510.01546/) (2025)
- [FlowCast: Trajectory Forecasting for Scalable Zero-Cost Speculative Flow Matching](https://mlsanity.com/paper/2602.01329/) (2026)
- [FastFlow: Accelerating The Generative Flow Matching Models with Bandit Inference](https://mlsanity.com/paper/2602.11105/) (2026)
- [Exploring the latent space of diffusion models directly through singular value decomposition](https://mlsanity.com/paper/2502.02225/) (2025)
- [Autoregressive Styled Text Image Generation, but Make it Reliable](https://mlsanity.com/paper/2510.23240/) (2025)
- [Fair Text to Medical Image Diffusion Model with Subgroup Distribution Aligned Tuning](https://mlsanity.com/paper/2406.14847/) (2024)
- [StylusAI: Stylistic Adaptation for Robust German Handwritten Text Generation](https://mlsanity.com/paper/2407.15608/) (2024)

## Abstract (arXiv)

Current text recognition systems, including those for handwritten scripts and scene text, have relied heavily on image synthesis and augmentation, since it is difficult to realize real-world complexity and diversity through collecting and annotating enough real text images. In this paper, we explore the problem of text image generation, by taking advantage of the powerful abilities of Diffusion Models in generating photo-realistic and diverse image samples with given conditions, and propose a method called Conditional Text Image Generation with Diffusion Models (CTIG-DM for short). To conform to the characteristics of text images, we devise three conditions: image condition, text condition, and style condition, which can be used to control the attributes, contents, and styles of the samples in the image generation process. Specifically, four text image generation modes, namely: (1) synthesis mode, (2) augmentation mode, (3) recovery mode, and (4) imitation mode, can be derived by combining and configuring these three conditions. Extensive experiments on both handwritten and scene text demonstrate that the proposed CTIG-DM is able to produce image samples that simulate real-world complexity and diversity, and thus can boost the performance of existing text recognizers. Besides, CTIG-DM shows its appealing potential in domain adaptation and generating images containing Out-Of-Vocabulary (OOV) words.