# Training Mixed-Domain Translation Models via Federated Learning

arXiv: https://arxiv.org/abs/2205.01557 · HTML: https://mlsanity.com/paper/2205.01557/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [A Survey of Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1806.00258/) (2018)
- [Federated Learning Of Out-Of-Vocabulary Words](https://mlsanity.com/paper/1903.10635/) (2019)
- [MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets](https://mlsanity.com/paper/1811.03850/) (2018)
- [Learning Private Neural Language Modeling with Attentive Aggregation](https://mlsanity.com/paper/1812.07108/) (2018)
- [Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation](https://mlsanity.com/paper/2101.00787/) (2021)
- [GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning](https://mlsanity.com/paper/2105.00529/) (2021)
- [Federated User Representation Learning](https://mlsanity.com/paper/1909.12535/) (2019)
- [Federated Learning for Ranking Browser History Suggestions](https://mlsanity.com/paper/1911.11807/) (2019)
- [FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learning](https://mlsanity.com/paper/2003.09288/) (2020)

## Built on by (most cited first, 12 of 12)

- [Tunable Soft Prompts are Messengers in Federated Learning](https://mlsanity.com/paper/2311.06805/) (2023)
- [Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter](https://mlsanity.com/paper/2305.12449/) (2023)
- [DP-NMT: Scalable Differentially-Private Machine Translation](https://mlsanity.com/paper/2311.14465/) (2023)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning](https://mlsanity.com/paper/2212.05789/) (2022)
- [Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarios](https://mlsanity.com/paper/2304.01005/) (2023)
- [Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation](https://mlsanity.com/paper/2401.07456/) (2024)
- [Personalized Federated Learning via Gradient Modulation for Heterogeneous Text Summarization](https://mlsanity.com/paper/2304.11524/) (2023)
- [A Comprehensive View of Personalized Federated Learning on Heterogeneous Clinical Datasets](https://mlsanity.com/paper/2309.16825/) (2023)
- [Fingerprint Attack: Client De-Anonymization in Federated Learning](https://mlsanity.com/paper/2310.05960/) (2023)
- [FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization](https://mlsanity.com/paper/2409.00116/) (2024)
- [Adaptive Latent-Space Constraints in Personalized Federated Learning](https://mlsanity.com/paper/2505.07525/) (2025)

## Abstract (arXiv)

Training mixed-domain translation models is a complex task that demands tailored architectures and costly data preparation techniques. In this work, we leverage federated learning (FL) in order to tackle the problem. Our investigation demonstrates that with slight modifications in the training process, neural machine translation (NMT) engines can be easily adapted when an FL-based aggregation is applied to fuse different domains. Experimental results also show that engines built via FL are able to perform on par with state-of-the-art baselines that rely on centralized training techniques. We evaluate our hypothesis in the presence of five datasets with different sizes, from different domains, to translate from German into English and discuss how FL and NMT can mutually benefit from each other. In addition to providing benchmarking results on the union of FL and NMT, we also propose a novel technique to dynamically control the communication bandwidth by selecting impactful parameters during FL updates. This is a significant achievement considering the large size of NMT engines that need to be exchanged between FL parties.