# UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

arXiv: https://arxiv.org/abs/2610.09823 · HTML: https://mlsanity.com/paper/2610.09823/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 34 of its 75 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 30 of 34)

- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [PaddleOCR 3.0 Technical Report](https://mlsanity.com/paper/2507.05595/) (2025)
- [X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again](https://mlsanity.com/paper/2507.22058/) (2025)
- [GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generation](https://mlsanity.com/paper/2303.17870/) (2023)
- [Qwen-Image-2.0 Technical Report](https://mlsanity.com/paper/2605.10730/) (2026)
- [TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation](https://mlsanity.com/paper/2502.07870/) (2025)
- [UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation](https://mlsanity.com/paper/2510.18701/) (2025)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation](https://mlsanity.com/paper/2605.28091/) (2026)
- [LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesis](https://mlsanity.com/paper/2503.21749/) (2025)
- [JoyType: A Robust Design for Multilingual Visual Text Creation](https://mlsanity.com/paper/2409.17524/) (2024)
- [TokBench: Evaluating Your Visual Tokenizer before Visual Generation](https://mlsanity.com/paper/2505.18142/) (2025)
- [OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities](https://mlsanity.com/paper/2507.15085/) (2025)
- [BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation](https://mlsanity.com/paper/2603.25732/) (2026)
- [WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editing](https://mlsanity.com/paper/2603.11593/) (2026)
- [Efficient Generative Model Training via Embedded Representation Warmup](https://mlsanity.com/paper/2504.10188/) (2025)
- [TextEditBench: Evaluating Reasoning-aware Text Editing Beyond Rendering](https://mlsanity.com/paper/2512.16270/) (2025)
- [TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models](https://mlsanity.com/paper/2411.02437/) (2024)
- [Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation](https://mlsanity.com/paper/2609.11638/) (2026)
- [LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes](https://mlsanity.com/paper/2609.03796/) (2026)
- [TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment](https://mlsanity.com/paper/2512.09350/) (2025)
- [Duality Models: An Embarrassingly Simple One-step Generation Paradigm](https://mlsanity.com/paper/2602.17682/) (2026)
- [Self-Adversarial One Step Generation via Condition Shifting](https://mlsanity.com/paper/2604.12322/) (2026)
- [Three-Body Scattering for Generative Modeling](https://mlsanity.com/paper/2607.18198/) (2026)
- [GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors](https://mlsanity.com/paper/2609.02349/) (2026)
- [TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images](https://mlsanity.com/paper/2603.07119/) (2026)
- [TextSculptor: Training and Benchmarking Scene Text Editing](https://mlsanity.com/paper/2605.21090/) (2026)

## Abstract (arXiv)

Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.