# Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens

arXiv: https://arxiv.org/abs/2305.11550 · HTML: https://mlsanity.com/paper/2305.11550/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1809.00357/) (2018)
- [Complete Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.10239/) (2020)
- [Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages](https://mlsanity.com/paper/2203.01976/) (2022)
- [Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations](https://mlsanity.com/paper/2004.14923/) (2020)
- [From Bilingual to Multilingual Neural Machine Translation by Incremental Training](https://mlsanity.com/paper/1907.00735/) (2019)

## Built on by (most cited first, 15 of 15)

- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer](https://mlsanity.com/paper/2402.14279/) (2024)
- [Multilingual k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2310.14644/) (2023)
- [A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation](https://mlsanity.com/paper/2403.20157/) (2024)
- [Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2406.08092/) (2024)
- [Disentangling the Roles of Target-Side Transfer and Regularization in Multilingual Machine Translation](https://mlsanity.com/paper/2402.01772/) (2024)
- [Investigating the translation capabilities of Large Language Models trained on parallel data only](https://mlsanity.com/paper/2406.09140/) (2024)
- [UvA-MT's Participation in the WMT23 General Translation Shared Task](https://mlsanity.com/paper/2310.09946/) (2023)
- [The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation](https://mlsanity.com/paper/2505.13090/) (2025)
- [Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2412.02101/) (2024)
- [Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2501.02979/) (2025)
- [Analyzing and Improving Cross-lingual Knowledge Transfer for Machine Translation](https://mlsanity.com/paper/2601.04036/) (2026)
- [The Impact of Vocabulary Overlaps on Knowledge Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2605.04196/) (2026)
- [Cross-Lingual Transfer for Machine Translation in Turkic Languages](https://mlsanity.com/paper/2607.29355/) (2026)
- [One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography](https://mlsanity.com/paper/2608.25904/) (2026)

## Abstract (arXiv)

We argue that translation quality alone is not a sufficient metric for measuring knowledge transfer in multilingual neural machine translation. To support this claim, we introduce Representational Transfer Potential (RTP), which measures representational similarities between languages. We show that RTP can measure both positive and negative transfer (interference), and find that RTP is strongly correlated with changes in translation quality, indicating that transfer does occur. Furthermore, we investigate data and language characteristics that are relevant for transfer, and find that multi-parallel overlap is an important yet under-explored feature. Based on this, we develop a novel training scheme, which uses an auxiliary similarity loss that encourages representations to be more invariant across languages by taking advantage of multi-parallel data. We show that our method yields increased translation quality for low- and mid-resource languages across multiple data and model setups.