# Dynamic Context-guided Capsule Network for Multimodal Machine Translation

arXiv: https://arxiv.org/abs/2009.02016 · HTML: https://mlsanity.com/paper/2009.02016/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2020.

## Built on by (most cited first, 30 of 36)

- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models](https://mlsanity.com/paper/2109.03415/) (2021)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [MSCTD: A Multimodal Sentiment Chat Translation Dataset](https://mlsanity.com/paper/2202.13645/) (2022)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation](https://mlsanity.com/paper/2407.02894/) (2024)
- [Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translation](https://mlsanity.com/paper/2205.01133/) (2022)
- [Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation](https://mlsanity.com/paper/2210.04468/) (2022)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [Learning with Capsules: A Survey](https://mlsanity.com/paper/2206.02664/) (2022)
- [Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective](https://mlsanity.com/paper/2210.08478/) (2022)
- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [M2C: Towards Automatic Multimodal Manga Complement](https://mlsanity.com/paper/2310.17130/) (2023)
- [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://mlsanity.com/paper/2310.13361/) (2023)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Low-resource Neural Machine Translation with Cross-modal Alignment](https://mlsanity.com/paper/2210.06716/) (2022)
- [Multimodal Neural Machine Translation with Search Engine Based Image Retrieval](https://mlsanity.com/paper/2208.00767/) (2022)
- [Adding Multimodal Capabilities to a Text-only Translation Model](https://mlsanity.com/paper/2403.03045/) (2024)
- [Video-Helpful Multimodal Machine Translation](https://mlsanity.com/paper/2310.20201/) (2023)
- [Towards Zero-Shot Multimodal Machine Translation](https://mlsanity.com/paper/2407.13579/) (2024)
- [IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task](https://mlsanity.com/paper/2107.01656/) (2021)
- [Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering](https://mlsanity.com/paper/2309.04734/) (2023)
- [TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries](https://mlsanity.com/paper/2505.05714/) (2025)
- [Dual-branch Prompting for Multimodal Machine Translation](https://mlsanity.com/paper/2507.17588/) (2025)

## Abstract (arXiv)

Multimodal machine translation (MMT), which mainly focuses on enhancing text-only translation with visual features, has attracted considerable attention from both computer vision and natural language processing communities. Most current MMT models resort to attention mechanism, global context modeling or multimodal joint representation learning to utilize visual features. However, the attention mechanism lacks sufficient semantic interactions between modalities while the other two provide fixed visual context, which is unsuitable for modeling the observed variability when generating translation. To address the above issues, in this paper, we propose a novel Dynamic Context-guided Capsule Network (DCCN) for MMT. Specifically, at each timestep of decoding, we first employ the conventional source-target attention to produce a timestep-specific source-side context vector. Next, DCCN takes this vector as input and uses it to guide the iterative extraction of related visual features via a context-guided dynamic routing mechanism. Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities. Finally, we obtain two multimodal context vectors, which are fused and incorporated into the decoder for the prediction of the target word. Experimental results on the Multi30K dataset of English-to-German and English-to-French translation demonstrate the superiority of DCCN. Our code is available on https://github.com/DeepLearnXMU/MM-DCCN.