# Communication-Efficient Federated Learning for Neural Machine Translation

arXiv: https://arxiv.org/abs/2112.06135 · HTML: https://mlsanity.com/paper/2112.06135/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 12 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [Practical Secure Aggregation for Federated Learning on User-Held Data](https://mlsanity.com/paper/1611.04482/) (2016)
- [FedVision: An Online Visual Object Detection Platform Powered by Federated Learning](https://mlsanity.com/paper/2001.06202/) (2020)
- [Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation](https://mlsanity.com/paper/2101.00787/) (2021)
- [Distributed Training of Deep Learning Models: A Taxonomic Perspective](https://mlsanity.com/paper/2007.03970/) (2020)
- [Two-stage Federated Phenotyping and Patient Representation Learning](https://mlsanity.com/paper/1908.05596/) (2019)

## Built on by (most cited first, 5 of 5)

- [Training Mixed-Domain Translation Models via Federated Learning](https://mlsanity.com/paper/2205.01557/) (2022)
- [Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter](https://mlsanity.com/paper/2305.12449/) (2023)
- [DP-NMT: Scalable Differentially-Private Machine Translation](https://mlsanity.com/paper/2311.14465/) (2023)
- [Continual Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2212.09097/) (2022)
- [Adaptive Parameterization of Deep Learning Models for Federated Learning](https://mlsanity.com/paper/2302.02949/) (2023)

## Abstract (arXiv)

Training neural machine translation (NMT) models in federated learning (FL) settings could be inefficient both computationally and communication-wise, due to the large size of translation engines as well as the multiple rounds of updates required to train clients and a central server. In this paper, we explore how to efficiently build NMT models in an FL setup by proposing a novel solution. In order to reduce the communication overhead, out of all neural layers we only exchange what we term "Controller" layers. Controllers are a small number of additional neural components connected to our pre-trained architectures. These new components are placed in between original layers. They act as liaisons to communicate with the central server and learn minimal information that is sufficient enough to update clients. We evaluated the performance of our models on five datasets from different domains to translate from German into English. We noted that the models equipped with Controllers preform on par with those trained in a central and non-FL setting. In addition, we observed a substantial reduction in the communication traffic of the FL pipeline, which is a direct consequence of using Controllers. Based on our experiments, Controller-based models are ~6 times less expensive than their other peers. This reduction is significantly important when we consider the number of parameters in large models and it becomes even more critical when such parameters need to be exchanged for multiple rounds in FL settings.