Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a promising distributed machine learning paradigm that enables multiple clients to collaboratively train a global model. In this paper, we focus on a practical federated multilingual learning setup where clients with their own language-specific data aim to collaboratively construct a high-quality neural machine translation (NMT) model. However, communication constraints in practical network systems present challenges for exchanging large-scale NMT engines between FL parties. We propose a meta-learning-based adaptive parameter selection methodology, MetaSend, that improves the communication efficiency of model transmissions from clients during FL-based multilingual NMT training. Our approach learns a dynamic threshold for filtering parameters prior to transmission without compromising the NMT model quality, based on the tensor deviations of clients between different FL rounds. Through experiments on two NMT datasets with different language distributions, we demonstrate that MetaSend obtains substantial improvements over baselines in translation quality in the presence of a limited communication budget.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 15 Jan 2024.
Builds on
42 of 64 references.
References · 42
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
- 2021Deduplicating Training Data Makes Language Models Bettercited by 705
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
30 more
- 2020Beyond English-Centric Multilingual Machine Translationcited by 559
- 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
- 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
- 2021Federated Learning on Non-IID Data: A Surveycited by 345
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
- 2019Revealing the Dark Secrets of BERTcited by 300
- 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
- 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
- 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
- 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
- 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
- 2020Compression of Deep Learning Models for Text: A Surveycited by 64
- 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
- 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
- 2020Methods for Pruning Deep Neural Networkscited by 49
- 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 42
- 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
- 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
- 2022Pretrained Models for Multilingual Federated Learningcited by 30
- 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
- 2021Plot and Rework: Modeling Storylines for Visual Storytellingcited by 14
- 2022Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learningcited by 13
- 2023Connectivity-Aware Semi-Decentralized Federated Learning over Time-Varying D2D Networkscited by 13
- 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12
- 2022Federated Learning with Flexible Controlcited by 12
- 2019Multi-hop Federated Private Data Augmentation with Sample Compressioncited by 4
- 2021Click-Based Student Performance Prediction: A Clustering Guided Meta-Learning Approachcited by 4
- 2020Distantly Supervised Relation Extraction in Federated Settingscited by 3
Built on by
Citing papers per year: 2024: 2.
2 papers.