# IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

arXiv: https://arxiv.org/abs/2603.10495 · HTML: https://mlsanity.com/paper/2603.10495/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 55 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Seedream 3.0 Technical Report](https://mlsanity.com/paper/2504.11346/) (2025)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Harmonizing Visual Representations for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.21979/) (2025)
- [Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model](https://mlsanity.com/paper/2505.23606/) (2025)
- [ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement](https://mlsanity.com/paper/2504.01934/) (2025)
- [TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation](https://mlsanity.com/paper/2505.05422/) (2025)
- [Unified Multimodal Discrete Diffusion](https://mlsanity.com/paper/2503.20853/) (2025)
- [FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities](https://mlsanity.com/paper/2505.20147/) (2025)
- [TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control](https://mlsanity.com/paper/2410.10133/) (2024)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending](https://mlsanity.com/paper/2410.10168/) (2024)

## Built on by (most cited first, 2 of 2)

- [MULTITEXTEDIT: Benchmarking Cross-Lingual Degradation in Text-in-Image Editing](https://mlsanity.com/paper/2605.08163/) (2026)
- [UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation](https://mlsanity.com/paper/2606.24333/) (2026)

## Abstract (arXiv)

End-to-end In-Image Machine Translation (IIMT) aims to convert text embedded within an image into a target language while preserving the original visual context, layout, and rendering style. However, existing IIMT benchmarks are largely synthetic and thus fail to reflect real-world complexity, while current evaluation protocols focus on single-modality metrics and overlook cross-modal faithfulness between rendered text and model outputs. To address these shortcomings, we present In-image Machine Translation Benchmark (IMTBench), a new benchmark of 2,500 image translation samples covering four practical scenarios and nine languages. IMTBench supports multi-aspect evaluation, including translation quality, background preservation, overall image quality, and a cross-modal alignment score that measures consistency between the translated text produced by the model and the text rendered in the translated image. We benchmark strong commercial cascade systems, and both closed- and open-source unified multi-modal models, and observe large performance gaps across scenarios and languages, especially on natural scenes and resource-limited languages, highlighting substantial headroom for end-to-end image text translation. We hope IMTBench establishes a standardized benchmark to accelerate progress in this emerging task.