# TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control

arXiv: https://arxiv.org/abs/2410.10133 · HTML: https://mlsanity.com/paper/2410.10133/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 60 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023)
- [Cross-Image Attention for Zero-Shot Appearance Transfer](https://mlsanity.com/paper/2311.03335/) (2023)
- [GlyphControl: Glyph Conditional Control for Visual Text Generation](https://mlsanity.com/paper/2305.18259/) (2023)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)

## Built on by (most cited first, 30 of 45)

- [FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing](https://mlsanity.com/paper/2505.03329/) (2025)
- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering](https://mlsanity.com/paper/2602.20903/) (2026)
- [TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis](https://mlsanity.com/paper/2505.17778/) (2025)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [Arbitrary Reading Order Scene Text Spotter with Local Semantics Guidance](https://mlsanity.com/paper/2412.10159/) (2024)
- [GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing](https://mlsanity.com/paper/2505.04915/) (2025)
- [VidText: Towards Comprehensive Evaluation for Video Text Understanding](https://mlsanity.com/paper/2505.22810/) (2025)
- [Rethinking Layered Graphic Design Generation with a Top-Down Approach](https://mlsanity.com/paper/2507.05601/) (2025)
- [Edicho: Consistent Image Editing in the Wild](https://mlsanity.com/paper/2412.21079/) (2024)
- [TextSSR: Diffusion-based Data Synthesis for Scene Text Recognition](https://mlsanity.com/paper/2412.01137/) (2024)
- [Type-R: Automatically Retouching Typos for Text-to-Image Generation](https://mlsanity.com/paper/2411.18159/) (2024)
- [Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2503.18746/) (2025)
- [Uni-DocDiff: A Unified Document Restoration Model Based on Diffusion](https://mlsanity.com/paper/2508.04055/) (2025)
- [StyleTextGen: Style-Conditioned Multilingual Scene Text Generation](https://mlsanity.com/paper/2605.14708/) (2026)
- [OmniText: A Training-Free Generalist for Controllable Text-Image Manipulation](https://mlsanity.com/paper/2510.24093/) (2025)
- [DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategy](https://mlsanity.com/paper/2512.01302/) (2025)
- [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026)
- [Class-Agnostic Region-of-Interest Matching in Document Images](https://mlsanity.com/paper/2506.21055/) (2025)
- [TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment](https://mlsanity.com/paper/2512.09350/) (2025)
- [VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents](https://mlsanity.com/paper/2602.00122/) (2026)
- [What Is Wrong with Synthetic Data for Scene Text Recognition? A Strong Synthetic Engine with Diverse Simulations and Self-Evolution](https://mlsanity.com/paper/2602.06450/) (2026)
- [IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation](https://mlsanity.com/paper/2603.10495/) (2026)
- [TextWand: A Unified Framework for Scene Text Editing](https://mlsanity.com/paper/2606.05730/) (2026)
- [InnoText: A Unified Model for Visual Text Generation and Editing](https://mlsanity.com/paper/2607.22101/) (2026)
- [TextRefine: Improving Textual Fidelity, Spatial Placement, and Glyph Rendering for Text Editing in Product Posters](https://mlsanity.com/paper/2608.19637/) (2026)
- [GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors](https://mlsanity.com/paper/2609.02349/) (2026)
- [SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Design](https://mlsanity.com/paper/2511.13285/) (2025)
- [Towards Training-Free Scene Text Editing](https://mlsanity.com/paper/2603.24571/) (2026)

## Abstract (arXiv)

Centred on content modification and style preservation, Scene Text Editing (STE) remains a challenging task despite considerable progress in text-to-image synthesis and text-driven image manipulation recently. GAN-based STE methods generally encounter a common issue of model generalization, while Diffusion-based STE methods suffer from undesired style deviations. To address these problems, we propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text glyph structure representation, TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. (ii) To further leverage the style prior, a Glyph-adaptive Mutual Self-attention mechanism is proposed which deconstructs the implicit fine-grained features of the source image to enhance style consistency and vision quality during inference. Furthermore, to fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons. Experiments demonstrate the effectiveness of TextCtrl compared with previous methods concerning both style fidelity and text accuracy.