mlsanityData to 14 Oct 2026

What Makes Pre-trained Language Models Better Zero-shot Learners?

arXiv 2209.15206 ↗First posted 30 Sep 2022cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Current methods for prompt learning in zeroshot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a posteriori. This is not ideal because in a realworld zero-shot scenario of practical relevance, no labelled data is available. Thus, we propose a simple yet effective method for screening reasonable prompt templates in zero-shot text classification: Perplexity Selection (Perplection). We hypothesize that language discrepancy can be used to measure the efficacy of prompt templates, and thereby develop a substantiated perplexity-based scheme allowing for forecasting the performance of prompt templates in advance. Experiments show that our method leads to improved prediction performance in a realistic zero-shot setting, eliminating the need for any labelled examples.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

26 of 39 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
14 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2021GPT Understands, Toocited by 716
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  11. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  12. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  13. 2022A Rationale-Centric Framework for Human-in-the-loop Machine Learningcited by 27
  14. 2021NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Predictioncited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 11; 2025: 5; 2026: 4.

23 papers.

Sort:

By area:

  1. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  2. 2025Advancing Sequential Numerical Prediction in Autoregressive Modelscited by 26
  3. 2024CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Modelscited by 23
  4. 2024Small Language Models are Good Too: An Empirical Study of Zero-Shot Classificationcited by 18
  5. 2024VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimizationcited by 16
  6. 2025Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performancecited by 13
  7. 2024When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?cited by 12
  8. 2024Reward Difference Optimization For Sample Reweighting In Offline RLHFcited by 10
  9. 2022SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learningcited by 8
  10. 2024Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Modelscited by 8
  11. 2024ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalnesscited by 7
  12. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
11 more
  1. 2023Large Language Models are In-context Teachers for Knowledge Reasoningcited by 4
  2. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoningcited by 4
  3. 2024Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Modelscited by 3
  4. 2025daDPO: Distribution-Aware DPO for Distilling Conversational Abilitiescited by 3
  5. 2025ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuningcited by 3
  6. 2024Can Separators Improve Chain-of-Thought Prompting?cited by 2
  7. 2026HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agentscited by 2
  8. 2026Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Modelscited by 2
  9. 2026Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuningcited by 2
  10. 2024SimCT: A Simple Consistency Test Protocol in LLMs Development Lifecyclecited by 1
  11. 2026TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuningcited by 0