# How Control Information Influences Multilingual Text Image Generation and Editing? arXiv: https://arxiv.org/abs/2407.11502 ยท HTML: https://mlsanity.com/paper/2407.11502/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 37 of its 40 references that are in the map. - First posted on arXiv 16 Jul 2024. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 37) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023) - [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023) - [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022) - [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023) - [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023) - [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021) - [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023) - [ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)](https://mlsanity.com/paper/1909.07145/) (2019) - [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023) - [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019) - [GlyphControl: Glyph Conditional Control for Visual Text Generation](https://mlsanity.com/paper/2305.18259/) (2023) - [ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard](https://mlsanity.com/paper/1912.09641/) (2019) - [ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT](https://mlsanity.com/paper/1909.07741/) (2019) - [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022) - [Editing Text in the Wild](https://mlsanity.com/paper/1908.03047/) (2019) - [Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering](https://mlsanity.com/paper/2403.09622/) (2024) - [SwapText: Image Based Texts Transfer in Scenes](https://mlsanity.com/paper/2003.08152/) (2020) - [STEFANN: Scene Text Editor using Font Adaptive Neural Network](https://mlsanity.com/paper/1903.01192/) (2019) - [Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach](https://mlsanity.com/paper/2011.14021/) (2020) - [DiffUTE: Universal Text Editing Diffusion Model](https://mlsanity.com/paper/2305.10825/) (2023) - [Exploring Stroke-Level Modifications for Scene Text Editing](https://mlsanity.com/paper/2212.01982/) (2022) ## Built on by (most cited first, 9 of 9) - [PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Rendering](https://mlsanity.com/paper/2504.06632/) (2025) - [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025) - [IGD: Instructional Graphic Design with Multimodal Layer Generation](https://mlsanity.com/paper/2507.09910/) (2025) - [TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards](https://mlsanity.com/paper/2605.19320/) (2026) - [Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generation](https://mlsanity.com/paper/2501.05892/) (2025) - [InnoText: A Unified Model for Visual Text Generation and Editing](https://mlsanity.com/paper/2607.22101/) (2026) - [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026) - [SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild](https://mlsanity.com/paper/2501.02962/) (2025) - [POCA: Pareto-Optimal Curriculum Alignment for Visual Text Generation](https://mlsanity.com/paper/2604.24171/) (2026) ## Abstract (arXiv) Visual text generation has significantly advanced through diffusion models aimed at producing images with readable and realistic text. Recent works primarily use a ControlNet-based framework, employing standard font text images to control diffusion models. Recognizing the critical role of control information in generating high-quality text, we investigate its influence from three perspectives: input encoding, role at different stages, and output features. Our findings reveal that: 1) Input control information has unique characteristics compared to conventional inputs like Canny edges and depth maps. 2) Control information plays distinct roles at different stages of the denoising process. 3) Output control features significantly differ from the base and skip features of the U-Net decoder in the frequency domain. Based on these insights, we propose TextGen, a novel framework designed to enhance generation quality by optimizing control information. We improve input and output features using Fourier analysis to emphasize relevant information and reduce noise. Additionally, we employ a two-stage generation framework to align the different roles of control information at different stages. Furthermore, we introduce an effective and lightweight dataset for training. Our method achieves state-of-the-art performance in both Chinese and English text generation. The code and dataset available at https://github.com/CyrilSterling/TextGen.