# UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editing arXiv: https://arxiv.org/abs/2601.08321 ยท HTML: https://mlsanity.com/paper/2601.08321/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 37 references that are in the map. - First posted on arXiv 13 Jan 2026. ## Builds on (most cited first, 28 of 28) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024) - [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025) - [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025) - [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025) - [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025) - [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023) - [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023) - [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023) - [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022) - [Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation](https://mlsanity.com/paper/2505.05472/) (2025) - [Character-Aware Models Improve Visual Text Rendering](https://mlsanity.com/paper/2212.10562/) (2022) - [Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering](https://mlsanity.com/paper/2403.09622/) (2024) - [UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models](https://mlsanity.com/paper/2312.04884/) (2023) - [AnyText2: Visual Text Generation and Editing With Customizable Attributes](https://mlsanity.com/paper/2411.15245/) (2024) - [Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Rendering](https://mlsanity.com/paper/2406.10208/) (2024) - [FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing](https://mlsanity.com/paper/2505.03329/) (2025) - [Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach](https://mlsanity.com/paper/2011.14021/) (2020) - [DiffUTE: Universal Text Editing Diffusion Model](https://mlsanity.com/paper/2305.10825/) (2023) - [DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models](https://mlsanity.com/paper/2503.01645/) (2025) - [CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encoder](https://mlsanity.com/paper/2412.17225/) (2024) - [UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis](https://mlsanity.com/paper/2507.00992/) (2025) ## Built on by (most cited first, 5 of 5) - [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026) - [GMO-E2DIT: Grounded Multi-Operation Editing for E-Commerce Images](https://mlsanity.com/paper/2607.00920/) (2026) - [TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy](https://mlsanity.com/paper/2603.09217/) (2026) - [Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering](https://mlsanity.com/paper/2609.34178/) (2026) - [Efficient Scene-Text Refinement via Pixel-Space Grouped Image Generation](https://mlsanity.com/paper/2610.18284/) (2026) ## Abstract (arXiv) With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM-Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.