mlsanityData to 7 Oct 2026

Multimodal Pivots for Image Caption Translation

arXiv 1601.03916 ↗First posted 15 Jan 2016cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

We present an approach to improve statistical machine translation of image descriptions by multimodal pivots defined in visual space. The key idea is to perform image retrieval over a database of images that are captioned in the target language, and use the captions of the most similar images for crosslingual reranking of translation outputs. Our approach does not depend on the availability of large amounts of in-domain parallel data, but only relies on available large datasets of monolingually captioned images, and on state-of-the-art convolutional neural networks to compute image similarities. Our experimental evaluation shows improvements of 1 BLEU point over strong baselines.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jan 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  2. 2021Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checkingcited by 52
  3. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  4. 2021Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfercited by 17
  5. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  6. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  7. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translationcited by 8
  8. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7
  9. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  10. 2024Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Conceptscited by 5
  11. 2021FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptionscited by 4
  12. 2022GLAMI-1M: A Multilingual Image-Text Fashion Datasetcited by 3
6 more
  1. 2023A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptionscited by 2
  2. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  3. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1
  4. 2024Image captioning in different languagescited by 0
  5. 2025A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioningcited by 0
  6. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0