mlsanityData to 7 Oct 2026

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

arXiv 2210.08478 ↗First posted 16 Oct 2022cs.CV cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 16 Oct 2022.

Builds on

33 of 41 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2018An efficient framework for learning sentence representationscited by 197
  8. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  9. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  10. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  11. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  12. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
21 more
  1. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  2. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  3. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  4. 2017Imagination improves Multimodal Translationcited by 38
  5. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  6. 2022On Vision Features in Multimodal Machine Translationcited by 31
  7. 2019Distilling Translations with Visual Awarenesscited by 30
  8. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  9. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  10. 2018The MeMAD Submission to the WMT18 Multimodal Translation Taskcited by 22
  11. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  12. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  13. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  14. 2022MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspectivecited by 17
  15. 2018CUNI System for the WMT18 Multimodal Translation Taskcited by 13
  16. 2020Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Groundingcited by 13
  17. 2018Large Margin Neural Language Modelcited by 12
  18. 2021Frequency-Aware Contrastive Learning for Neural Machine Translationcited by 11
  19. 2017An empirical study on the effectiveness of images in Multimodal Neural Machine Translationcited by 10
  20. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  21. 2021Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Informationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

10 papers.

  1. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  2. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 13
  3. 2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6
  4. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  5. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  6. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  7. 2025Information-Theoretic Complementary Prompts for Improved Continual Text Classificationcited by 0
  8. 2025Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detectioncited by 0
  9. 2025Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cuescited by 0
  10. 2026SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learningcited by 0