# Co-training Improves Prompt-based Learning for Large Language Models

arXiv: https://arxiv.org/abs/2202.00828 · HTML: https://mlsanity.com/paper/2202.00828/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [Reordering Examples Helps during Priming-based Few-Shot Learning](https://mlsanity.com/paper/2106.01751/) (2021)
- [Self-Training with Weak Supervision](https://mlsanity.com/paper/2104.05514/) (2021)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training](https://mlsanity.com/paper/1909.00415/) (2019)

## Built on by (most cited first, 24 of 24)

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Synthetic continued pretraining](https://mlsanity.com/paper/2409.07431/) (2024)
- [Training Strategies for Efficient Embodied Reasoning](https://mlsanity.com/paper/2505.08243/) (2025)
- [PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions](https://mlsanity.com/paper/2305.14908/) (2023)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Theoretical Analysis of Weak-to-Strong Generalization](https://mlsanity.com/paper/2405.16043/) (2024)
- [LangNav: Language as a Perceptual Representation for Navigation](https://mlsanity.com/paper/2310.07889/) (2023)
- [How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?](https://mlsanity.com/paper/2305.01555/) (2023)
- [Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model](https://mlsanity.com/paper/2210.02498/) (2022)
- [Training Subset Selection for Weak Supervision](https://mlsanity.com/paper/2206.02914/) (2022)
- [Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts](https://mlsanity.com/paper/2210.03690/) (2022)
- [Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency](https://mlsanity.com/paper/2305.10713/) (2023)
- [Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting](https://mlsanity.com/paper/2310.18804/) (2023)
- [Zero-Label Prompt Selection](https://mlsanity.com/paper/2211.04668/) (2022)
- [Entailment as Robust Self-Learner](https://mlsanity.com/paper/2305.17197/) (2023)
- [A Pilot Study on Dialogue-Level Dependency Parsing for Chinese](https://mlsanity.com/paper/2305.12441/) (2023)
- [Leveraging Large Language Models for Structure Learning in Prompted Weak Supervision](https://mlsanity.com/paper/2402.01867/) (2024)
- [VIKSER: Visual Knowledge-Driven Self-Reinforcing Reasoning Framework](https://mlsanity.com/paper/2502.00711/) (2025)
- [Regularized Data Programming with Automated Bayesian Prior Selection](https://mlsanity.com/paper/2210.08677/) (2022)
- [Differentiable Instruction Optimization for Cross-Task Generalization](https://mlsanity.com/paper/2306.10098/) (2023)
- [Student as an Inherent Denoiser of Noisy Teacher](https://mlsanity.com/paper/2312.10185/) (2023)
- [GPT-4 Generated Narratives of Life Events using a Structured Narrative Prompt: A Validation Study](https://mlsanity.com/paper/2402.05435/) (2024)

## Abstract (arXiv)

We demonstrate that co-training (Blum & Mitchell, 1998) can improve the performance of prompt-based learning by using unlabeled data. While prompting has emerged as a promising paradigm for few-shot and zero-shot learning, it is often brittle and requires much larger models compared to the standard supervised setup. We find that co-training makes it possible to improve the original prompt model and at the same time learn a smaller, downstream task-specific model. In the case where we only have partial access to a prompt model (e.g., output probabilities from GPT-3 (Brown et al., 2020)) we learn a calibration model over the prompt outputs. When we have full access to the prompt model's gradients but full finetuning remains prohibitively expensive (e.g., T0 (Sanh et al., 2021)), we learn a set of soft prompt continuous vectors to iteratively update the prompt model. We find that models trained in this manner can significantly improve performance on challenging datasets where there is currently a large gap between prompt-based learning and fully-supervised models.