# Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training arXiv: https://arxiv.org/abs/2108.11119 ยท HTML: https://mlsanity.com/paper/2108.11119/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 50 references that are in the map. - First posted on arXiv 25 Aug 2021. ## Builds on (most cited first, 30 of 39) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019) - [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014) - [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019) - [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019) - [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016) - [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019) - [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017) - [Learning Fashion Compatibility with Bidirectional LSTMs](https://mlsanity.com/paper/1707.05691/) (2017) - [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021) - [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020) - [Doubly-Attentive Decoder for Multi-modal Neural Machine Translation](https://mlsanity.com/paper/1702.01287/) (2017) - [Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback](https://mlsanity.com/paper/1905.12794/) (2019) - [Incorporating Global Visual Features into Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1701.06521/) (2017) - [Imagination improves Multimodal Translation](https://mlsanity.com/paper/1705.04350/) (2017) - [Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards](https://mlsanity.com/paper/2008.02693/) (2020) - [Dynamic Context-guided Capsule Network for Multimodal Machine Translation](https://mlsanity.com/paper/2009.02016/) (2020) - [Attention Strategies for Multi-Source Sequence-to-Sequence Learning](https://mlsanity.com/paper/1704.06567/) (2017) - [Distilling Translations with Visual Awareness](https://mlsanity.com/paper/1906.07701/) (2019) - [Visual Grounding in Video for Unsupervised Word Translation](https://mlsanity.com/paper/2003.05078/) (2020) - [A Visual Attention Grounding Neural Model for Multimodal Machine Translation](https://mlsanity.com/paper/1808.08266/) (2018) ## Built on by (most cited first, 10 of 10) - [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023) - [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022) - [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022) - [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022) - [MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2508.11999/) (2025) - [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024) - [Generalizing Multimodal Pre-training into Multilingual via Language Acquisition](https://mlsanity.com/paper/2206.11091/) (2022) - [GLAMI-1M: A Multilingual Image-Text Fashion Dataset](https://mlsanity.com/paper/2211.14451/) (2022) - [ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT](https://mlsanity.com/paper/2506.04929/) (2025) - [Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion](https://mlsanity.com/paper/2602.21646/) (2026) ## Abstract (arXiv) Translating e-commercial product descriptions, a.k.a product-oriented machine translation (PMT), is essential to serve e-shoppers all over the world. However, due to the domain specialty, the PMT task is more challenging than traditional machine translation problems. Firstly, there are many specialized jargons in the product description, which are ambiguous to translate without the product image. Secondly, product descriptions are related to the image in more complicated ways than standard image descriptions, involving various visual aspects such as objects, shapes, colors or even subjective styles. Moreover, existing PMT datasets are small in scale to support the research. In this paper, we first construct a large-scale bilingual product description dataset called Fashion-MMT, which contains over 114k noisy and 40k manually cleaned description translations with multiple product images. To effectively learn semantic alignments among product images and bilingual texts in translation, we design a unified product-oriented cross-modal cross-lingual model (upoc~) for pre-training and fine-tuning. Experiments on the Fashion-MMT and Multi30k datasets show that our model significantly outperforms the state-of-the-art models even pre-trained on the same dataset. It is also shown to benefit more from large-scale noisy data to improve the translation quality. We will release the dataset and codes at https://github.com/syuqings/Fashion-MMT.