# UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

arXiv: https://arxiv.org/abs/2606.24333 · HTML: https://mlsanity.com/paper/2606.24333/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 30 of 35)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Harmonizing Visual Representations for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.21979/) (2025)
- [Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model](https://mlsanity.com/paper/2505.23606/) (2025)
- [ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement](https://mlsanity.com/paper/2504.01934/) (2025)
- [TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation](https://mlsanity.com/paper/2505.05422/) (2025)
- [Unified Multimodal Discrete Diffusion](https://mlsanity.com/paper/2503.20853/) (2025)
- [FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities](https://mlsanity.com/paper/2505.20147/) (2025)
- [TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control](https://mlsanity.com/paper/2410.10133/) (2024)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding](https://mlsanity.com/paper/2503.14140/) (2025)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending](https://mlsanity.com/paper/2410.10168/) (2024)
- [Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation](https://mlsanity.com/paper/2407.02894/) (2024)
- [Exploring In-Image Machine Translation with Real-World Background](https://mlsanity.com/paper/2505.15282/) (2025)
- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)

## Built on by (most cited first, 1 of 1)

- [TransAnyText: Translating Arbitrary Text in E-commerce Images via Structured Visual Generation](https://mlsanity.com/paper/2608.16284/) (2026)

## Abstract (arXiv)

In-Image Machine Translation (IIMT) aims to translate scene text in an image and render the translated text back into the original regions while preserving the overall visual appearance. Recent unified multimodal models provide a promising solution by combining visual-text understanding and image generation within a single framework. However, directly adapting such models to IIMT remains challenging. In particular, they often suffer from understanding-generation conflicts, where the translation inferred during understanding is inconsistent with the text supervision used in generation, and spatial position misalignment, where the rendered text does not accurately match the target text regions. To address these issues, we present UniTranslator, a unified multimodal framework for IIMT that tightly couples translation understanding and text editing. Specifically, we introduce an Understand-Generation Alignment Module (UGAM) to bridge the representation gap between understanding and generation, encouraging semantic consistency between translated content prediction and text rendering. We further propose a Spatial Mask Decoder (SMD) with pixel-level supervision over text regions to improve spatial grounding, geometric alignment, and layout controllability during generation. Extensive experiments on multiple benchmarks demonstrate that UniTranslator achieves state-of-the-art performance across diverse language directions and complex real-world layouts. Moreover, our results reveal a strong mutual reinforcement effect between translation understanding and image generation, highlighting the advantage of unified translation multimodal learning. Code is available at https://github.com/SeerRay-Lab/Unitranslator.