# Parameter Sharing Methods for Multilingual Self-Attentional Translation Models

arXiv: https://arxiv.org/abs/1809.00252 · HTML: https://mlsanity.com/paper/1809.00252/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2018.

## Built on by (most cited first, 30 of 33)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Benchmarking Differential Privacy and Federated Learning for BERT Models](https://mlsanity.com/paper/2106.13973/) (2021)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2212.10551/) (2022)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Robust Optimization for Multilingual Translation with Imbalanced Data](https://mlsanity.com/paper/2104.07639/) (2021)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.15236/) (2022)
- [Learning Language-Specific Layers for Multilingual Machine Translation](https://mlsanity.com/paper/2305.02665/) (2023)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Adaptive Sparse Transformer for Multilingual Translation](https://mlsanity.com/paper/2104.07358/) (2021)
- [MTVR: Multilingual Moment Retrieval in Videos](https://mlsanity.com/paper/2108.00061/) (2021)
- [Breaking Down Multilingual Machine Translation](https://mlsanity.com/paper/2110.08130/) (2021)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning](https://mlsanity.com/paper/2202.05451/) (2022)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [Parameter Differentiation based Multilingual Neural Machine Translation](https://mlsanity.com/paper/2112.13619/) (2021)
- [Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders](https://mlsanity.com/paper/2102.06578/) (2021)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)
- [Hierarchical Transformer for Multilingual Machine Translation](https://mlsanity.com/paper/2103.03589/) (2021)
- [Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model](https://mlsanity.com/paper/2205.15544/) (2022)
- [One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging](https://mlsanity.com/paper/2604.02881/) (2026)
- [Informative Language Representation Learning for Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.01530/) (2022)
- [Searching for Effective Multilingual Fine-Tuning Methods: A Case Study in Summarization](https://mlsanity.com/paper/2212.05740/) (2022)
- [TPDM: Selectively Removing Positional Information for Zero-shot Translation via Token-Level Position Disentangle Module](https://mlsanity.com/paper/2305.19857/) (2023)
- [An Empirical Study of Automatic Post-Editing](https://mlsanity.com/paper/2209.07759/) (2022)

## Abstract (arXiv)

In multilingual neural machine translation, it has been shown that sharing a single translation model between multiple languages can achieve competitive performance, sometimes even leading to performance gains over bilingually trained models. However, these improvements are not uniform; often multilingual parameter sharing results in a decrease in accuracy due to translation models not being able to accommodate different languages in their limited parameter space. In this work, we examine parameter sharing techniques that strike a happy medium between full sharing and individual training, specifically focusing on the self-attentional Transformer model. We find that the full parameter sharing approach leads to increases in BLEU scores mainly when the target languages are from a similar language family. However, even in the case where target languages are from different families where full parameter sharing leads to a noticeable drop in BLEU scores, our proposed methods for partial sharing of parameters can lead to substantial improvements in translation accuracy.