mlsanityData to 14 Oct 2026

Towards Practical Few-shot Federated NLP

arXiv 2212.00192 ↗First posted 1 Dec 2022cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

29 of 46 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
17 more
  1. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  2. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  3. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  4. 2021How Many Data Points is a Prompt Worth?cited by 198
  5. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 155
  6. 2018Deep Learning for Sentiment Analysis : A Surveycited by 139
  7. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  8. 2020Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspectivecited by 103
  9. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  10. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  11. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 75
  12. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  13. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  14. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  15. 2021Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementationcited by 40
  16. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  17. 2021Federated Few-Shot Learning with Adversarial Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 0; 2026: 1.

5 papers.

  1. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 26
  2. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  3. 2023MixPro: Simple yet Effective Data Augmentation for Prompt-based Learningcited by 4
  4. 2024PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examplescited by 3
  5. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3