mlsanityData to 14 Oct 2026

Imagination improves Multimodal Translation

arXiv 1705.04350 ↗First posted 11 May 2017cs.CL cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 8; 2024: 4; 2025: 7; 2026: 3.

38 papers.

Sort:

By area:

  1. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  2. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  3. 2022On Vision Features in Multimodal Machine Translationcited by 31
  4. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  5. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  6. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  7. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
  8. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  9. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  10. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  11. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  12. 2021A Survey on Low-Resource Neural Machine Translationcited by 16
26 more
  1. 2022Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translationcited by 13
  2. 2023Large Scale Multi-Lingual Multi-Modal Summarization Datasetcited by 12
  3. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  4. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  5. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  6. 2024AnyTrans: Translate AnyText in the Image with Large Scale Modelscited by 8
  7. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 8
  8. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7
  9. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  10. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  11. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  12. 2025Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translationcited by 6
  13. 20243AM: An Ambiguity-Aware Multi-Modal Machine Translation Datasetcited by 4
  14. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  15. 2024Towards Zero-Shot Multimodal Machine Translationcited by 3
  16. 2022Supervised Visual Attention for Simultaneous Multimodal Machine Translationcited by 2
  17. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  18. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  19. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  20. 2025Multimodal Machine Translation with Visual Scene Graph Pruningcited by 1
  21. 2023Modality Influence in Multimodal Machine Learningcited by 0
  22. 2023Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generationcited by 0
  23. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Taskscited by 0
  24. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0
  25. 2026Comparative Evaluation of Machine Translation Systems on Images with Textcited by 0
  26. 2026TransMeme: A Multi-Agent Framework for Cross-Cultural Meme Transcreationcited by 0