# Show Me the World in My Language: Establishing the First Baseline for Scene-Text to Scene-Text Translation

arXiv: https://arxiv.org/abs/2308.03024 · HTML: https://mlsanity.com/paper/2308.03024/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 6 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Image Quality Assessment using Contrastive Learning](https://mlsanity.com/paper/2110.13266/) (2021)
- [Editing Text in the Wild](https://mlsanity.com/paper/1908.03047/) (2019)
- [SwapText: Image Based Texts Transfer in Scenes](https://mlsanity.com/paper/2003.08152/) (2020)
- [TextStyleBrush: Transfer of Text Aesthetics from a Single Example](https://mlsanity.com/paper/2106.08385/) (2021)
- [STEFANN: Scene Text Editor using Font Adaptive Neural Network](https://mlsanity.com/paper/1903.01192/) (2019)
- [Exploring Stroke-Level Modifications for Scene Text Editing](https://mlsanity.com/paper/2212.01982/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Towards End-to-End In-Image Neural Machine Translation](https://mlsanity.com/paper/2010.10648/) (2020)
- [Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task](https://mlsanity.com/paper/2210.03887/) (2022)
- [MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning](https://mlsanity.com/paper/1911.09483/) (2019)
- [Benchmarking Scene Text Recognition in Devanagari, Telugu and Malayalam](https://mlsanity.com/paper/2104.04437/) (2021)
- [RewriteNet: Reliable Scene Text Editing with Implicit Decomposition of Text Contents and Styles](https://mlsanity.com/paper/2107.11041/) (2021)

## Built on by (most cited first, 5 of 5)

- [Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processing](https://mlsanity.com/paper/2402.03082/) (2024)
- [UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation](https://mlsanity.com/paper/2606.24333/) (2026)
- [Accurate Scene Text Recognition with Efficient Model Scaling and Cloze Self-Distillation](https://mlsanity.com/paper/2503.16184/) (2025)
- [Global-Local Aware Scene Text Editing](https://mlsanity.com/paper/2512.03574/) (2025)
- [TextDS: Parameter-Efficient Representation Alignment for Scene Text Detection under Distribution Shifts](https://mlsanity.com/paper/2606.28077/) (2026)

## Abstract (arXiv)

In this work, we study the task of ``visually'' translating scene text from a source language (e.g., Hindi) to a target language (e.g., English). Visual translation involves not just the recognition and translation of scene text but also the generation of the translated image that preserves visual features of the source scene text, such as font, size, and background. There are several challenges associated with this task, such as translation with limited context, deciding between translation and transliteration, accommodating varying text lengths within fixed spatial boundaries, and preserving the font and background styles of the source scene text in the target language. To address this problem, we make the following contributions: (i) We study visual translation as a standalone problem for the first time in the literature. (ii) We present a cascaded framework for visual translation that combines state-of-the-art modules for scene text recognition, machine translation, and scene text synthesis as a baseline for the task. (iii) We propose a set of task-specific design enhancements to design a variant of the baseline to obtain performance improvements. (iv) Currently, the existing related literature lacks any comprehensive performance evaluation for this novel task. To fill this gap, we introduce several automatic and user-assisted evaluation metrics designed explicitly for evaluating visual translation. Further, we evaluate presented baselines for translating scene text between Hindi and English. Our experiments demonstrate that although we can effectively perform visual translation over a large collection of scene text images, the presented baseline only partially addresses challenges posed by visual translation tasks. We firmly believe that this new task and the limitations of existing models, as reported in this paper, should encourage further research in visual translation.