# Does Multimodality Help Human and Machine for Translation and Image Captioning?

arXiv: https://arxiv.org/abs/1605.09186 · HTML: https://mlsanity.com/paper/1605.09186/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2016.

## Built on by (most cited first, 7 of 7)

- [Cross-lingual Visual Pre-training for Multimodal Machine Translation](https://mlsanity.com/paper/2101.10044/) (2021)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [Detecting Concrete Visual Tokens for Multimodal Machine Translation](https://mlsanity.com/paper/2403.03075/) (2024)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)
- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)
- [Multimodal Machine Translation with Visual Scene Graph Pruning](https://mlsanity.com/paper/2505.19507/) (2025)
- [Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translation](https://mlsanity.com/paper/2504.18012/) (2025)

## Abstract (arXiv)

This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural networks models trained using monomodal or multimodal data. We also performed a human evaluation in order to estimate the usefulness of multimodal data for human machine translation and image description generation. Our systems obtained the best results for both tasks according to the automatic evaluation metrics BLEU and METEOR.