# Multimodal Machine Translation with Visual Scene Graph Pruning

arXiv: https://arxiv.org/abs/2505.19507 · HTML: https://mlsanity.com/paper/2505.19507/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020)
- [Doubly-Attentive Decoder for Multi-modal Neural Machine Translation](https://mlsanity.com/paper/1702.01287/) (2017)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Incorporating Global Visual Features into Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1701.06521/) (2017)
- [Imagination improves Multimodal Translation](https://mlsanity.com/paper/1705.04350/) (2017)
- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [Scene Graph Parsing as Dependency Parsing](https://mlsanity.com/paper/1803.09189/) (2018)
- [Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models](https://mlsanity.com/paper/2109.03415/) (2021)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Does Multimodality Help Human and Machine for Translation and Image Captioning?](https://mlsanity.com/paper/1605.09186/) (2016)
- [ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation](https://mlsanity.com/paper/2211.04861/) (2022)
- [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://mlsanity.com/paper/2310.13361/) (2023)
- [CUNI System for WMT16 Automatic Post-Editing and Multimodal Translation Tasks](https://mlsanity.com/paper/1606.07481/) (2016)
- [Adding Multimodal Capabilities to a Text-only Translation Model](https://mlsanity.com/paper/2403.03045/) (2024)

## Built on by (most cited first, 1 of 1)

- [ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning](https://mlsanity.com/paper/2507.07306/) (2025)

## Abstract (arXiv)

Multimodal machine translation (MMT) seeks to address the challenges posed by linguistic polysemy and ambiguity in translation tasks by incorporating visual information. A key bottleneck in current MMT research is the effective utilization of visual data. Previous approaches have focused on extracting global or region-level image features and using attention or gating mechanisms for multimodal information fusion. However, these methods have not adequately tackled the issue of visual information redundancy in MMT, nor have they proposed effective solutions. In this paper, we introduce a novel approach--multimodal machine translation with visual Scene Graph Pruning (PSG), which leverages language scene graph information to guide the pruning of redundant nodes in visual scene graphs, thereby reducing noise in downstream translation tasks. Through extensive comparative experiments with state-of-the-art methods and ablation studies, we demonstrate the effectiveness of the PSG model. Our results also highlight the promising potential of visual information pruning in advancing the field of MMT.