mlsanityData to 7 Oct 2026

Neural Machine Translation with Phrase-Level Universal Visual Representations

arXiv 2203.10299 ↗First posted 19 Mar 2022cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them suffer from shortage of sentence-image pairs. In this paper, we propose a phrase-level retrieval-based method for MMT to get visual information for the source input from existing sentence-image data sets so that MMT can break the limitation of paired sentence-image input. Our method performs retrieval at the phrase level and hence learns visual information from pairs of source phrase and grounded region, which can mitigate data sparsity. Furthermore, our method employs the conditional variational auto-encoder to learn visual representations which can filter redundant visual information and only retain visual information related to the phrase. Experiments show that the proposed method significantly outperforms strong baselines on multiple MMT datasets, especially when the textual context is limited.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 19 Mar 2022.

Builds on

42 of 58 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2015Generating Sentences from a Continuous Spacecited by 607
  12. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
30 more
  1. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  2. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  3. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  4. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  5. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  6. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  7. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  8. 2017Imagination improves Multimodal Translationcited by 38
  9. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  10. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  11. 2019Distilling Translations with Visual Awarenesscited by 30
  12. 2017Emergent Translation in Multi-Agent Communicationcited by 22
  13. 2018A Visual Attention Grounding Neural Model for Multimodal Machine Translationcited by 22
  14. 2020Generative Imagination Elevates Machine Translationcited by 22
  15. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  16. 2016Multimodal Attention for Neural Machine Translationcited by 19
  17. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  18. 2016Multimodal Pivots for Image Caption Translationcited by 18
  19. 2020Simultaneous Machine Translation with Visual Contextcited by 17
  20. 2018CUNI System for the WMT18 Multimodal Translation Taskcited by 13
  21. 2020Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Groundingcited by 13
  22. 2017Memory-augmented Neural Machine Translationcited by 12
  23. 2017Search Engine Guided Non-Parametric Neural Machine Translationcited by 10
  24. 2017An empirical study on the effectiveness of images in Multimodal Neural Machine Translationcited by 10
  25. 2021Mixup Decoding for Diverse Machine Translationcited by 10
  26. 2019Visual Agreement Regularized Training for Multi-Modal Machine Translationcited by 9
  27. 2017Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translationcited by 8
  28. 2018LIUM-CVC Submissions for WMT18 Multimodal Translation Taskcited by 8
  29. 2017LIUM-CVC Submissions for WMT17 Multimodal Translation Taskcited by 7
  30. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 11; 2024: 5; 2025: 3; 2026: 1.

26 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  4. 2023Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucinationcited by 27
  5. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  6. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  7. 2024On the Evaluation of Machine-Generated Reportscited by 23
  8. 2023Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignmentcited by 15
  9. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  10. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  11. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  12. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7
14 more
  1. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  2. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  3. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  4. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  5. 2024m3P: Towards Multimodal Multilingual Translation with Multimodal Promptcited by 5
  6. 2022Improving Variational Autoencoders with Density Gap-based Regularizationcited by 3
  7. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  8. 2023Video-Helpful Multimodal Machine Translationcited by 3
  9. 2023Enhancing Neural Machine Translation with Semantic Unitscited by 2
  10. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  11. 2025Multimodal Machine Translation with Visual Scene Graph Pruningcited by 1
  12. 2023Unified Model Learning for Various Neural Machine Translationcited by 0
  13. 2024MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translationcited by 0
  14. 2026Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weightingcited by 0