# Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task

arXiv: https://arxiv.org/abs/2210.03887 · HTML: https://mlsanity.com/paper/2210.03887/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 8 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [A Convolutional Encoder Model for Neural Machine Translation](https://mlsanity.com/paper/1611.02344/) (2016)
- [Towards Fully Automated Manga Translation](https://mlsanity.com/paper/2012.14271/) (2020)
- [Towards End-to-End In-Image Neural Machine Translation](https://mlsanity.com/paper/2010.10648/) (2020)

## Built on by (most cited first, 20 of 20)

- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation](https://mlsanity.com/paper/2407.02894/) (2024)
- [MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation](https://mlsanity.com/paper/2412.07147/) (2024)
- [AnyTrans: Translate AnyText in the Image with Large Scale Models](https://mlsanity.com/paper/2406.11432/) (2024)
- [Exploring In-Image Machine Translation with Real-World Background](https://mlsanity.com/paper/2505.15282/) (2025)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translation](https://mlsanity.com/paper/2305.05166/) (2023)
- [Show Me the World in My Language: Establishing the First Baseline for Scene-Text to Scene-Text Translation](https://mlsanity.com/paper/2308.03024/) (2023)
- [MT3: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2505.19714/) (2025)
- [Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency](https://mlsanity.com/paper/2507.08309/) (2025)
- [Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation](https://mlsanity.com/paper/2507.07572/) (2025)
- [MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation](https://mlsanity.com/paper/2603.23896/) (2026)
- [Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation](https://mlsanity.com/paper/2602.21956/) (2026)
- [Multi-Teacher Knowledge Distillation For Text Image Machine Translation](https://mlsanity.com/paper/2305.05226/) (2023)
- [MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation](https://mlsanity.com/paper/2604.16943/) (2026)
- [LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation](https://mlsanity.com/paper/2507.18940/) (2025)
- [Vectra: A New Metric, Dataset, and Model for Visual Quality Assessment in E-Commerce In-Image Machine Translation](https://mlsanity.com/paper/2602.07014/) (2026)
- [Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion](https://mlsanity.com/paper/2602.21646/) (2026)
- [VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation](https://mlsanity.com/paper/2605.24675/) (2026)
- [Comparative Evaluation of Machine Translation Systems on Images with Text](https://mlsanity.com/paper/2605.29476/) (2026)

## Abstract (arXiv)

End-to-end text image translation (TIT), which aims at translating the source language embedded in images to the target language, has attracted intensive attention in recent research. However, data sparsity limits the performance of end-to-end text image translation. Multi-task learning is a non-trivial way to alleviate this problem via exploring knowledge from complementary related tasks. In this paper, we propose a novel text translation enhanced text image translation, which trains the end-to-end model with text translation as an auxiliary task. By sharing model parameters and multi-task training, our model is able to take full advantage of easily-available large-scale text parallel corpus. Extensive experimental results show our proposed method outperforms existing end-to-end methods, and the joint multi-task learning with both text translation and recognition tasks achieves better results, proving translation and recognition auxiliary tasks are complementary.