mlsanityData to 14 Oct 2026

DiffUTE: Universal Text Editing Diffusion Model

arXiv 2305.10825 ↗First posted 18 May 2023cs.CVcited by 45

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in urlhttps://github.com/chenhaoxing/DiffUTE.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 45 of its 56 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

45 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoders12 Apr
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  5. 2022Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model26 Aug
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2022Galactica: A Large Language Model for Sciencecited by 744
25 more
  1. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  2. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  3. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  4. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  5. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  6. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  7. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  8. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 139
  9. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 128
  10. 2021Paint by Wordcited by 107
  11. 2022Character-Aware Models Improve Visual Text Renderingcited by 92
  12. 2019Editing Text in the Wildcited by 82
  13. 2018Spatial Fusion GAN for Image Synthesiscited by 68
  14. 2020SwapText: Image Based Texts Transfer in Scenescited by 50
  15. 2019STEFANN: Scene Text Editor using Font Adaptive Neural Networkcited by 47
  16. 2021SynthTIGER: Synthetic Text Image GEneratoR Towards Better Text Recognition Modelscited by 43
  17. 2022Exploring Stroke-Level Modifications for Scene Text Editingcited by 37
  18. 2022Look Closer to Supervise Better: One-Shot Font Generation via Component-Based Discriminatorcited by 32
  19. 2021De-rendering Stylized Textscited by 24
  20. 2022Hierarchical Dynamic Image Harmonizationcited by 12
  21. 2021RewriteNet: Reliable Scene Text Editing with Implicit Decomposition of Text Contents and Stylescited by 10
  22. 2022A2: Efficient Automated Attacker for Boosting Adversarial Trainingcited by 10
  23. 2022SCSNet: An Efficient Paradigm for Learning Simultaneously Image Colorization and Super-Resolutioncited by 8
  24. 2023Mobile User Interface Element Detection Via Adaptively Prompt Tuningcited by 8
  25. 2022GenText: Unsupervised Artistic Text Generation via Decoupled Font and Texture Manipulationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 14; 2026: 9.

45 papers.

Sort:

By area:

  1. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 170
  2. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  3. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 81
  4. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 59
  5. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  6. 2025PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Renderingcited by 46
  7. 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 45
  8. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 40
  9. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 20
  10. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 19
  11. 2024Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editingcited by 18
  12. 2024DreamText: High Fidelity Scene Text Synthesiscited by 18
33 more
  1. 2024CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encodercited by 14
  2. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  3. 2024Novel Object Synthesis via Adaptive Text-Image Harmonycited by 12
  4. 2025GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editingcited by 11
  5. 2025ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotationscited by 10
  6. 2024How Control Information Influences Multilingual Text Image Generation and Editing?cited by 9
  7. 2024Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generationcited by 9
  8. 2025Rethinking Machine Unlearning in Image Generation Modelscited by 9
  9. 2024Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processingcited by 8
  10. 2024TextSSR: Diffusion-based Data Synthesis for Scene Text Recognitioncited by 8
  11. 2023FASTER: A Font-Agnostic Scene Text Editing and Rendering Frameworkcited by 6
  12. 2024MetaDesigner: Advancing Artistic Typography Through AI-Driven, User-Centric, and Multilingual WordArt Synthesiscited by 6
  13. 2026InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generationcited by 6
  14. 2024Typographic Text Generation with Off-the-Shelf Diffusion Modelcited by 5
  15. 2026UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editingcited by 5
  16. 2024Conditional Prototype Rectification Prompt Learningcited by 3
  17. 2025Calligrapher: Freestyle Text Image Customizationcited by 3
  18. 2025OmniText: A Training-Free Generalist for Controllable Text-Image Manipulationcited by 3
  19. 2025MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detectioncited by 3
  20. 2026SteerVTE: Seamless Video Text Editing with Style and Glyph Controlcited by 3
  21. 2024FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Imagescited by 2
  22. 2026Edit-Based Refinement for Parallel Masked Diffusion Language Modelscited by 2
  23. 2023Segment Anything Model Meets Image Harmonizationcited by 1
  24. 2024The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learningcited by 1
  25. 2025HDGlyph: A Hierarchical Disentangled Glyph-Based Framework for Long-Tail Text Rendering in Diffusion Modelscited by 1
  26. 2025SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Designcited by 1
  27. 2026Towards Training-Free Scene Text Editingcited by 1
  28. 2025SceneTextStylizer: A Training-Free Scene Text Style Transfer Framework with Diffusion Modelcited by 0
  29. 2025TripleFDS: Triple Feature Disentanglement and Synthesis for Scene Text Editingcited by 0
  30. 2026Enhanced Video Text Editing with Trajectory-Aligned Glyph Renderingcited by 0
  31. 2026STEPS: Scene Text Editing with Preserved Style Using Diffusion and Contrastive Style Encodingcited by 0
  32. 2026ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editingcited by 0
  33. 2026Efficient Scene-Text Refinement via Pixel-Space Grouped Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning
  2. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance
  3. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing
  4. 2023On Manipulating Scene Text in the Wild with Diffusion Models
  5. 2023Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
  6. 2023Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models
  7. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering
  8. 2024TextMastero: Mastering High-Quality Scene Text Editing in Diverse Languages and Styles