# Multilingual NMT with a language-independent attention bridge

arXiv: https://arxiv.org/abs/1811.00498 · HTML: https://mlsanity.com/paper/1811.00498/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 20 of 20)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Democratizing Neural Machine Translation with OPUS-MT](https://mlsanity.com/paper/2212.01936/) (2022)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation](https://mlsanity.com/paper/2205.12216/) (2022)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [Attentive fine-tuning of Transformers for Translation of low-resourced languages @LoResMT 2021](https://mlsanity.com/paper/2108.08556/) (2021)
- [HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2207.04906/) (2022)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders](https://mlsanity.com/paper/2102.06578/) (2021)
- [Multilingual Entity and Relation Extraction from Unified to Language-specific Training](https://mlsanity.com/paper/2301.04434/) (2023)
- [Revisiting Language Encoding in Learning Multilingual Representations](https://mlsanity.com/paper/2102.08357/) (2021)
- [SIT at MixMT 2022: Fluent Translation Built on Giant Pre-trained Models](https://mlsanity.com/paper/2210.11670/) (2022)
- [Exploiting Multilingualism in Low-resource Neural Machine Translation via Adversarial Learning](https://mlsanity.com/paper/2303.18011/) (2023)
- [Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.10190/) (2023)
- [Multilingual Neural Machine Translation System for Indic to Indic Languages](https://mlsanity.com/paper/2306.12693/) (2023)
- [Enhancing Context Through Contrast](https://mlsanity.com/paper/2401.03314/) (2024)
- [I Have an Attention Bridge to Sell You: Generalization Capabilities of Modular Translation Architectures](https://mlsanity.com/paper/2404.17918/) (2024)
- [The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space](https://mlsanity.com/paper/2609.00515/) (2026)

## Abstract (arXiv)

In this paper, we propose a multilingual encoder-decoder architecture capable of obtaining multilingual sentence representations by means of incorporating an intermediate em attention bridge that is shared across all languages. That is, we train the model with language-specific encoders and decoders that are connected via self-attention with a shared layer that we call attention bridge. This layer exploits the semantics from each language for performing translation and develops into a language-independent meaning representation that can efficiently be used for transfer learning. We present a new framework for the efficient development of multilingual NMT using this model and scheduled training. We have tested the approach in a systematic way with a multi-parallel data set. We show that the model achieves substantial improvements over strong bilingual models and that it also works well for zero-shot translation, which demonstrates its ability of abstraction and transfer learning.