mlsanityData to 7 Oct 2026

Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

arXiv 1702.01287 ↗First posted 4 Feb 2017cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mechanisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 11; 2023: 9; 2024: 5; 2025: 6; 2026: 1.

46 papers.

Sort:

By area:

  1. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  2. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  3. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  4. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  5. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  6. 2021Semantic Representation for Dialogue Modelingcited by 24
  7. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  8. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  9. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  10. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  11. 2021A Survey on Low-Resource Neural Machine Translationcited by 16
  12. 2022MSCTD: A Multimodal Sentiment Chat Translation Datasetcited by 15
34 more
  1. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  2. 2022Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translationcited by 13
  3. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  4. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  5. 2024AnyTrans: Translate AnyText in the Image with Large Scale Modelscited by 8
  6. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7
  7. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  8. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  9. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  10. 2021ViTA: Visual-Linguistic Translation by Aligning Object Tagscited by 5
  11. 2021The JDDC 2.0 Corpus: A Large-Scale Multimodal Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Servicecited by 5
  12. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  13. 2022Multimodal Neural Machine Translation with Search Engine Based Image Retrievalcited by 4
  14. 20243AM: An Ambiguity-Aware Multi-Modal Machine Translation Datasetcited by 4
  15. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  16. 2021Visual Cues and Error Correction for Translation Robustnesscited by 2
  17. 2021IITP at WAT 2021: System description for English-Hindi Multimodal Translation Taskcited by 2
  18. 2022Supervised Visual Attention for Simultaneous Multimodal Machine Translationcited by 2
  19. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  20. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  21. 2021Narration Generation for Cartoon Videoscited by 1
  22. 2021ICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarizationcited by 1
  23. 2023Few-shot Domain-Adaptive Visually-fused Event Detection from Textcited by 1
  24. 2024Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasetscited by 1
  25. 2024Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioningcited by 1
  26. 2025Multimodal Machine Translation with Visual Scene Graph Pruningcited by 1
  27. 2025ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MTcited by 1
  28. 2025GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translationcited by 1
  29. 2023Generalization algorithm of multimodal pre-training model based on graph-text self-supervised trainingcited by 0
  30. 2023Accessible Instruction-Following Agentcited by 0
  31. 2023CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarizationcited by 0
  32. 2025Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translationcited by 0
  33. 2025A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translationcited by 0
  34. 2026Video-guided Machine Translation with Global Video Contextcited by 0