mlsanityData to 7 Oct 2026

Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

arXiv 2210.14803 ↗First posted 26 Oct 2022cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

28 of 47 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
16 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  6. 2021True Few-Shot Learning with Language Modelscited by 343
  7. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  10. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  11. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  12. 2021Generating Datasets with Pretrained Language Modelscited by 192
  13. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  14. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  15. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  16. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 3.

13 papers.

Sort:

By area:

  1. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  2. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  3. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  4. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  5. 2023Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Taskscited by 21
  6. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  7. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  8. 2023Zero-Shot Text Classification via Self-Supervised Tuningcited by 6
  9. 2024FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schemacited by 6
  10. 2023MixPro: Simple yet Effective Data Augmentation for Prompt-based Learningcited by 4
  11. 2023The Benefits of Label-Description Training for Zero-Shot Text Classificationcited by 4
  12. 2023Knowledge Rumination for Pre-trained Language Modelscited by 2
1 more
  1. 2023WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categoriescited by 2