# Hierarchical Transformer for Multilingual Machine Translation

arXiv: https://arxiv.org/abs/2103.03589 · HTML: https://mlsanity.com/paper/2103.03589/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://mlsanity.com/paper/1801.07772/) (2018)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Parameter Sharing Methods for Multilingual Self-Attentional Translation Models](https://mlsanity.com/paper/1809.00252/) (2018)
- [A Framework for Hierarchical Multilingual Machine Translation](https://mlsanity.com/paper/2005.05507/) (2020)

## Built on by (most cited first, 2 of 2)

- [Deep Learning Models in Software Requirements Engineering](https://mlsanity.com/paper/2105.07771/) (2021)
- [Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation](https://mlsanity.com/paper/2509.17930/) (2025)

## Abstract (arXiv)

The choice of parameter sharing strategy in multilingual machine translation models determines how optimally parameter space is used and hence, directly influences ultimate translation quality. Inspired by linguistic trees that show the degree of relatedness between different languages, the new general approach to parameter sharing in multilingual machine translation was suggested recently. The main idea is to use these expert language hierarchies as a basis for multilingual architecture: the closer two languages are, the more parameters they share. In this work, we test this idea using the Transformer architecture and show that despite the success in previous work there are problems inherent to training such hierarchical models. We demonstrate that in case of carefully chosen training strategy the hierarchical architecture can outperform bilingual models and multilingual models with full parameter sharing.