# Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

arXiv: https://arxiv.org/abs/2210.14803 · HTML: https://mlsanity.com/paper/2210.14803/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks](https://mlsanity.com/paper/2009.08445/) (2020)
- [ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data](https://mlsanity.com/paper/2205.12600/) (2022)

## Built on by (most cited first, 13 of 13)

- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [Adapting Large Language Models to Domains via Reading Comprehension](https://mlsanity.com/paper/2309.09530/) (2023)
- [Controllable Mixed-Initiative Dialogue Generation through Prompting](https://mlsanity.com/paper/2305.04147/) (2023)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks](https://mlsanity.com/paper/2305.17653/) (2023)
- [Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching](https://mlsanity.com/paper/2406.06326/) (2024)
- [SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation](https://mlsanity.com/paper/2405.10040/) (2024)
- [Zero-Shot Text Classification via Self-Supervised Tuning](https://mlsanity.com/paper/2305.11442/) (2023)
- [FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema](https://mlsanity.com/paper/2402.11811/) (2024)
- [The Benefits of Label-Description Training for Zero-Shot Text Classification](https://mlsanity.com/paper/2305.02239/) (2023)
- [MixPro: Simple yet Effective Data Augmentation for Prompt-based Learning](https://mlsanity.com/paper/2304.09402/) (2023)
- [Knowledge Rumination for Pre-trained Language Models](https://mlsanity.com/paper/2305.08732/) (2023)
- [WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories](https://mlsanity.com/paper/2307.15293/) (2023)

## Abstract (arXiv)

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.