mlsanityData to 7 Oct 2026

Federated Few-Shot Learning for Mobile NLP

arXiv 2212.05974 ↗First posted 12 Dec 2022cs.LG cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications. For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FeS, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0×, 41.2× and 3000.0×, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 73 of its 112 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

73 of 112 references.

References · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2019Advances and Open Problems in Federated Learningcited by 3,021
  11. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
38 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  8. 2019Deep Leakage from Gradientscited by 1,004
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2019Towards Federated Learning at Scale: System Designcited by 826
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  13. 2019A Closer Look at Few-shot Classificationcited by 679
  14. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  15. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  16. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  17. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  18. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  19. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  20. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  21. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  22. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  23. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  24. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  25. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  26. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  27. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  28. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  29. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  30. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  31. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  32. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  33. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  34. 2018Deep Learning for Sentiment Analysis : A Surveycited by 139
  35. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  36. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  37. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  38. 2019One-Shot Federated Learningcited by 124

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 6; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  2. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  3. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  4. 2025FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clientscited by 7
  5. 2025Uncovering Gradient Inversion Risks in Practical Language Model Trainingcited by 6
  6. 2024FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmissioncited by 4
  7. 2024PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examplescited by 3
  8. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3
  9. 2024FedAli: Personalized Federated Learning Alignment with Prototype Layers for Generalized Mobile Servicescited by 2
  10. 2025Efficient Federated Fine-Tuning of Large Language Models with Layer Dropoutcited by 2
  11. 2025Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmissioncited by 2
  12. 2026Mitigating Gradient Inversion Risks in Language Models via Token Obfuscationcited by 2
3 more
  1. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  2. 2025PracMHBench: Re-evaluating Model-Heterogeneous Federated Learning Based on Practical Edge Device Constraintscited by 0
  3. 2026CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correctioncited by 0