# Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding

arXiv: https://arxiv.org/abs/2101.05208 · HTML: https://mlsanity.com/paper/2101.05208/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2020.

## Built on by (most cited first, 13 of 13)

- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Adding Multimodal Capabilities to a Text-only Translation Model](https://mlsanity.com/paper/2403.03045/) (2024)
- [Detecting Concrete Visual Tokens for Multimodal Machine Translation](https://mlsanity.com/paper/2403.03075/) (2024)
- [Dual-branch Prompting for Multimodal Machine Translation](https://mlsanity.com/paper/2507.17588/) (2025)
- [ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT](https://mlsanity.com/paper/2506.04929/) (2025)
- [Morphology-Aware KOA Classification: Integrating Graph Priors with Vision Models](https://mlsanity.com/paper/2510.21801/) (2025)
- [Leveraging Textual-Cues for Enhancing Multimodal Sentiment Analysis by Object Recognition](https://mlsanity.com/paper/2602.00360/) (2026)

## Abstract (arXiv)

Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that visual information is less explored in MMT as it is often redundant to textual information. In this paper, we propose an object-level visual context modeling framework (OVC) to efficiently capture and explore visual information for multimodal machine translation. With detected objects, the proposed OVC encourages MMT to ground translation on desirable visual objects by masking irrelevant objects in the visual modality. We equip the proposed with an additional object-masking loss to achieve this goal. The object-masking loss is estimated according to the similarity between masked objects and the source texts so as to encourage masking source-irrelevant objects. Additionally, in order to generate vision-consistent target words, we further propose a vision-weighted translation loss for OVC. Experiments on MMT datasets demonstrate that the proposed OVC model outperforms state-of-the-art MMT models and analyses show that masking irrelevant objects helps grounding in MMT.