# Exploring In-Image Machine Translation with Real-World Background

arXiv: https://arxiv.org/abs/2505.15282 · HTML: https://mlsanity.com/paper/2505.15282/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model](https://mlsanity.com/paper/2312.12232/) (2023)
- [OCR-VQGAN: Taming Text-within-Image Generation](https://mlsanity.com/paper/2210.11248/) (2022)
- [Towards End-to-End In-Image Neural Machine Translation](https://mlsanity.com/paper/2010.10648/) (2020)
- [Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task](https://mlsanity.com/paper/2210.03887/) (2022)
- [Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation](https://mlsanity.com/paper/2407.02894/) (2024)

## Built on by (most cited first, 7 of 7)

- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://mlsanity.com/paper/2601.07338/) (2026)
- [IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation](https://mlsanity.com/paper/2603.10495/) (2026)
- [UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation](https://mlsanity.com/paper/2606.24333/) (2026)
- [Vectra: A New Metric, Dataset, and Model for Visual Quality Assessment in E-Commerce In-Image Machine Translation](https://mlsanity.com/paper/2602.07014/) (2026)
- [Comparative Evaluation of Machine Translation Systems on Images with Text](https://mlsanity.com/paper/2605.29476/) (2026)
- [VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics](https://mlsanity.com/paper/2609.37287/) (2026)

## Abstract (arXiv)

In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenario IIMT, we design an IIMT dataset that includes subtitle text with real-world background. However previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on text-image directly, and fuses the translated text-image with the background, to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect.