mlsanityData to 14 Oct 2026

Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning

arXiv 2212.05789 ↗First posted 12 Dec 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The increasing privacy concerns on personal private text data promote the development of federated learning (FL) in recent years. However, the existing studies on applying FL in NLP are not suitable to coordinate participants with heterogeneous or private learning objectives. In this study, we further broaden the application scope of FL in NLP by proposing an Assign-Then-Contrast (denoted as ATC) framework, which enables clients with heterogeneous NLP tasks to construct an FL course and learn useful knowledge from each other. Specifically, the clients are suggested to first perform local training with the unified tasks assigned by the server rather than using their own learning objectives, which is called the Assign training stage. After that, in the Contrast training stage, clients train with different local learning objectives and exchange knowledge with other clients who contribute consistent and useful model updates. We conduct extensive experiments on six widely-used datasets covering both Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks, and the proposed ATC framework achieves significant improvements compared with various baseline methods. The source code is available at urlhttps://github.com/alibaba/FederatedScope/tree/master/federatedscope/nlp/heterotasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

34 of 46 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
22 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,049
  2. 2017Federated Multi-Task Learningcited by 672
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  4. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  5. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  6. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  7. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  10. 2020Adaptive Personalized Federated Learningcited by 358
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  13. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  14. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  15. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  16. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  17. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  18. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  19. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  20. 2020FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learningcited by 24
  21. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  22. 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  2. 2023Tunable Soft Prompts are Messengers in Federated Learningcited by 10
  3. 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Datacited by 1
  4. 2024FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalizationcited by 0