# LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners

arXiv: https://arxiv.org/abs/2110.06274 · HTML: https://mlsanity.com/paper/2110.06274/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [Learning to Self-Train for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1906.00562/) (2019)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [Multimodal Emergent Fake News Detection via Meta Neural Process Networks](https://mlsanity.com/paper/2106.13711/) (2021)
- [Learning from Language Description: Low-shot Named Entity Recognition via Decomposed Framework](https://mlsanity.com/paper/2109.05357/) (2021)

## Built on by (most cited first, 9 of 9)

- [Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts](https://mlsanity.com/paper/2210.11292/) (2022)
- [Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning](https://mlsanity.com/paper/2305.18170/) (2023)
- [Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement](https://mlsanity.com/paper/2312.15731/) (2023)
- [C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2505.17773/) (2025)
- [Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding](https://mlsanity.com/paper/2310.13022/) (2023)
- [Leveraging Natural Supervision for Language Representation Learning and Generation](https://mlsanity.com/paper/2207.10617/) (2022)
- [HugNLP: A Unified and Comprehensive Library for Natural Language Processing](https://mlsanity.com/paper/2302.14286/) (2023)
- [Improving Prompt Tuning with Learned Prompting Layers](https://mlsanity.com/paper/2310.20127/) (2023)
- [Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification](https://mlsanity.com/paper/2312.04982/) (2023)

## Abstract (arXiv)

We present a new method LiST is short for Lite Prompted Self-Training for parameter-efficient fine-tuning of large pre-trained language models (PLMs) for few-shot learning. LiST improves over recent methods that adopt prompt-based fine-tuning (FN) using two key techniques. The first is the use of self-training to leverage large amounts of unlabeled data for prompt-based FN in few-shot settings. We use self-training in conjunction with meta-learning for re-weighting noisy pseudo-prompt labels. Self-training is expensive as it requires updating all the model parameters repetitively. Therefore, we use a second technique for light-weight fine-tuning where we introduce a small number of task-specific parameters that are fine-tuned during self-training while keeping the PLM encoder frozen. Our experiments show that LiST can effectively leverage unlabeled data to improve the model performance for few-shot learning. Additionally, the fine-tuning is efficient as it only updates a small percentage of parameters and the overall model footprint is reduced since several tasks can share a common PLM encoder as backbone. A comprehensive study on six NLU tasks demonstrate LiST to improve by 35% over classic fine-tuning and 6% over prompt-based FN with 96% reduction in number of trainable parameters when fine-tuned with no more than 30 labeled examples from each task. With only 14M tunable parameters, LiST outperforms GPT-3 in-context learning by 33% on few-shot NLU tasks.