mlsanityData to 14 Oct 2026

Conditional Text Image Generation with Diffusion Models

arXiv 2306.10804 ↗First posted 19 Jun 2023cs.CVcited by 58

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Current text recognition systems, including those for handwritten scripts and scene text, have relied heavily on image synthesis and augmentation, since it is difficult to realize real-world complexity and diversity through collecting and annotating enough real text images. In this paper, we explore the problem of text image generation, by taking advantage of the powerful abilities of Diffusion Models in generating photo-realistic and diverse image samples with given conditions, and propose a method called Conditional Text Image Generation with Diffusion Models (CTIG-DM for short). To conform to the characteristics of text images, we devise three conditions: image condition, text condition, and style condition, which can be used to control the attributes, contents, and styles of the samples in the image generation process. Specifically, four text image generation modes, namely: (1) synthesis mode, (2) augmentation mode, (3) recovery mode, and (4) imitation mode, can be derived by combining and configuring these three conditions. Extensive experiments on both handwritten and scene text demonstrate that the proposed CTIG-DM is able to produce image samples that simulate real-world complexity and diversity, and thus can boost the performance of existing text recognizers. Besides, CTIG-DM shows its appealing potential in domain adaptation and generating images containing Out-Of-Vocabulary (OOV) words.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 53 of its 77 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

53 of 77 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  3. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  4. 2020Diffusion models for Handwriting Generation13 Nov

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
37 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  2. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  3. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 520
  4. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  5. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 307
  6. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  7. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  8. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  9. 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
  10. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  11. 2014Reading Text in the Wild with Convolutional Neural Networkscited by 122
  12. 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
  13. 2018Scene Text Detection and Recognition: The Deep Learning Eracited by 114
  14. 2019Decoupled Attention Network for Text Recognitioncited by 102
  15. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  16. 2020RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognitioncited by 85
  17. 2018ESIR: End-to-end Scene Text Recognition via Iterative Image Rectificationcited by 79
  18. 2020ScrabbleGAN: Semi-Supervised Varying Length Handwritten Text Generationcited by 69
  19. 2019TextScanner: Reading Characters in Order for Robust Scene Text Recognitioncited by 68
  20. 2020GANwriting: Content-Conditioned Generation of Styled Handwritten Word Imagescited by 68
  21. 2021What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labelscited by 63
  22. 2018Geometry Score: A Method For Comparing Generative Adversarial Networkscited by 57
  23. 2015Reading Scene Text in Deep Convolutional Sequencescited by 44
  24. 2020UnrealText: Synthesizing Realistic Scene Text Images from the Unreal Worldcited by 43
  25. 2019LOGAN: Latent Optimisation for Generative Adversarial Networkscited by 42
  26. 2019Adversarial Generation of Handwritten Text Images Conditioned on Sequencescited by 37
  27. 2021Handwriting Transformerscited by 37
  28. 2018Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognitioncited by 36
  29. 2020Learn to Augment: Joint Data Augmentation and Network Optimization for Text Recognitioncited by 36
  30. 2022Multi-Granularity Prediction for Scene Text Recognitioncited by 36
  31. 2020Text and Style Conditioned GAN for Generation of Offline Handwriting Linescited by 35
  32. 2018Handwriting Recognition in Low-resource Scripts using Adversarial Learningcited by 32
  33. 2022Look Closer to Supervise Better: One-Shot Font Generation via Component-Based Discriminatorcited by 32
  34. 2020Text Recognition in the Wild: A Surveycited by 29
  35. 2021MetaHTR: Towards Writer-Adaptive Handwritten Text Recognitioncited by 27
  36. 2022SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Textcited by 26
  37. 2022Levenshtein OCRcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 19; 2025: 23; 2026: 15.

58 papers.

Sort:

By area:

  1. 2023FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learningcited by 48
  2. 2024Diffusion-Based Neural Network Weights Generationcited by 39
  3. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  4. 2024Diffusion Model is a Good Pose Estimator from 3D RF-Visioncited by 22
  5. 2024Beyond Deepfake Images: Detecting AI-Generated Videoscited by 22
  6. 2024Visual Text Generation in the Wildcited by 22
  7. 2024One-Shot Diffusion Mimicker for Handwritten Text Generationcited by 20
  8. 2024VATr++: Choose Your Words Wisely for Handwritten Text Generationcited by 18
  9. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  10. 2025DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Modecited by 17
  11. 2024DiffusionPen: Towards Controlling the Style of Handwritten Text Generationcited by 16
  12. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 13
38 more
  1. 2025Zero-Shot Styled Text Image Generation, but Make It Autoregressivecited by 13
  2. 2025Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generationcited by 10
  3. 2025OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilitiescited by 9
  4. 2024Text Image Inpainting via Global Structure-Guided Diffusion Modelscited by 8
  5. 2024TextSSR: Diffusion-based Data Synthesis for Scene Text Recognitioncited by 8
  6. 2024E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Datacited by 7
  7. 2024Rethinking HTG Evaluation: Bridging Generation and Recognitioncited by 6
  8. 2025Diffusion Model Quantization: A Reviewcited by 6
  9. 2024Typographic Text Generation with Off-the-Shelf Diffusion Modelcited by 5
  10. 2024Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Modelscited by 5
  11. 2025TRADES: Generating Realistic Market Simulations with Diffusion Modelscited by 5
  12. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  13. 2026FlowCast: Trajectory Forecasting for Scalable Zero-Cost Speculative Flow Matchingcited by 4
  14. 2026FastFlow: Accelerating The Generative Flow Matching Models with Bandit Inferencecited by 4
  15. 2025Exploring the latent space of diffusion models directly through singular value decompositioncited by 3
  16. 2025Autoregressive Styled Text Image Generation, but Make it Reliablecited by 3
  17. 2024Fair Text to Medical Image Diffusion Model with Subgroup Distribution Aligned Tuningcited by 2
  18. 2024StylusAI: Stylistic Adaptation for Robust German Handwritten Text Generationcited by 2
  19. 2024GroupDiff: Diffusion-based Group Portrait Editingcited by 2
  20. 2025D2-DPM: Dual Denoising for Quantized Diffusion Probabilistic Modelscited by 2
  21. 2025Quo Vadis Handwritten Text Generation for Handwritten Text Recognition?cited by 2
  22. 2025Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generationcited by 2
  23. 2026What Is Wrong with Synthetic Data for Scene Text Recognition? A Strong Synthetic Engine with Diverse Simulations and Self-Evolutioncited by 2
  24. 2025SceneVTG++: Controllable Multilingual Visual Text Generation in the Wildcited by 1
  25. 2025Physics-Guided Image Dehazing Diffusioncited by 1
  26. 2025Where's the liability in the Generative Era? Recovery-based Black-Box Detection of AI-Generated Contentcited by 1
  27. 2025Non-asymptotic convergence bound of conditional diffusion modelscited by 1
  28. 2025FW-GAN: Frequency-Driven Handwriting Synthesis with Wave-Modulated MLP Generatorcited by 1
  29. 2026CONSTANT: Towards High-Quality One-Shot Handwriting Generation with Patch Contrastive Enhancement and Style-Aware Quantizationcited by 1
  30. 2026T2LSC-Bench: Benchmarking Localized Semantic Control in Text-to-Image Generationcited by 1
  31. 2024Text Image Generation for Low-Resource Languages with Dual Translation Learningcited by 0
  32. 2024MCGM: Mask Conditional Text-to-Image Generative Modelcited by 0
  33. 2025DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusioncited by 0
  34. 2025VisualChef: Generating Visual Aids in Cooking via Mask Inpaintingcited by 0
  35. 2025FreeDNA: Endowing Domain Adaptation of Diffusion-Based Dense Prediction with Training-Free Domain Noise Alignmentcited by 0
  36. 2025Neural Bridge Processescited by 0
  37. 2026Unknown Aware AI-Generated Content Attributioncited by 0
  38. 2026Single-Stage Signal Attenuation Diffusion Model for Low-Light Image Enhancement and Denoisingcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation
  2. 2023Controlled and Conditional Text to Image Generation with Diffusion Prior
  3. 2024Conditional Text-to-Image Generation with Reference Guidance
  4. 2024Test-time Conditional Text-to-Image Synthesis Using Diffusion Models
  5. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models
  6. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  7. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  8. 2023Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models