mlsanityData to 14 Oct 2026

Exploring Better Text Image Translation with Multimodal Codebook

arXiv 2305.17415 ↗First posted 27 May 2023cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of applications and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing convenience for subsequent studies. Then, we propose a TIT model with a multimodal codebook, which is able to associate the image with relevant texts, providing useful supplementary information for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilingual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experiments and in-depth analyses strongly demonstrate the effectiveness of our proposed model and training framework.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  8. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  9. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  10. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 139
  11. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 105
  12. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
23 more
  1. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  2. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  3. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  4. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  5. 2017Imagination improves Multimodal Translationcited by 38
  6. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  7. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  8. 2022On Vision Features in Multimodal Machine Translationcited by 31
  9. 2019Distilling Translations with Visual Awarenesscited by 30
  10. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  11. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  12. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  13. 2018Input Combination Strategies for Multi-Source Transformer Decodercited by 22
  14. 2020Generative Imagination Elevates Machine Translationcited by 22
  15. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  16. 2019Chinese Street View Text: Large-scale Chinese Text Reading with Partially Supervised Learningcited by 20
  17. 2020Towards End-to-End In-Image Neural Machine Translationcited by 20
  18. 2022Improving End-to-End Text Image Translation From the Auxiliary Text Translation Taskcited by 20
  19. 2016Multimodal Attention for Neural Machine Translationcited by 19
  20. 2020Simultaneous Machine Translation with Visual Contextcited by 17
  21. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  22. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  23. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 7; 2026: 12.

23 papers.

Sort:

By area:

  1. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  2. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  3. 2023Show Me the World in My Language: Establishing the First Baseline for Scene-Text to Scene-Text Translationcited by 5
  4. 2025MT3: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learningcited by 5
  5. 2025Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translationcited by 4
  6. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translationcited by 4
  7. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptationcited by 3
  8. 2025ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoningcited by 3
  9. 2026Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translationcited by 3
  10. 2023Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filteringcited by 2
  11. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  12. 2025Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completioncited by 2
11 more
  1. 2026IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translationcited by 2
  2. 2026MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translationcited by 2
  3. 2026ICDAR 2025 Competition on End-to-End Document Image Machine Translation Towards Complex Layoutscited by 1
  4. 2026UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translationcited by 1
  5. 2025LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translationcited by 0
  6. 2026LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantizationcited by 0
  7. 2026Vectra: A New Metric, Dataset, and Model for Visual Quality Assessment in E-Commerce In-Image Machine Translationcited by 0
  8. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0
  9. 2026VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translationcited by 0
  10. 2026Comparative Evaluation of Machine Translation Systems on Images with Textcited by 0
  11. 2026VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubricscited by 0