# Federated Few-Shot Learning for Mobile NLP

arXiv: https://arxiv.org/abs/2212.05974 · HTML: https://mlsanity.com/paper/2212.05974/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 112 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 30 of 73)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)

## Built on by (most cited first, 15 of 15)

- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [Grounding Foundation Models through Federated Transfer Learning: A General Framework](https://mlsanity.com/paper/2311.17431/) (2023)
- [FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients](https://mlsanity.com/paper/2502.20639/) (2025)
- [Uncovering Gradient Inversion Risks in Practical Language Model Training](https://mlsanity.com/paper/2507.21198/) (2025)
- [FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission](https://mlsanity.com/paper/2403.00881/) (2024)
- [PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examples](https://mlsanity.com/paper/2403.06131/) (2024)
- [Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach](https://mlsanity.com/paper/2604.21197/) (2026)
- [FedAli: Personalized Federated Learning Alignment with Prototype Layers for Generalized Mobile Services](https://mlsanity.com/paper/2411.10595/) (2024)
- [Efficient Federated Fine-Tuning of Large Language Models with Layer Dropout](https://mlsanity.com/paper/2503.10217/) (2025)
- [Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission](https://mlsanity.com/paper/2507.00082/) (2025)
- [Mitigating Gradient Inversion Risks in Language Models via Token Obfuscation](https://mlsanity.com/paper/2602.15897/) (2026)
- [Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices](https://mlsanity.com/paper/2508.19078/) (2025)
- [PracMHBench: Re-evaluating Model-Heterogeneous Federated Learning Based on Practical Edge Device Constraints](https://mlsanity.com/paper/2509.08750/) (2025)
- [CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correction](https://mlsanity.com/paper/2601.12917/) (2026)

## Abstract (arXiv)

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications. For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FeS, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0×, 41.2× and 3000.0×, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled.