mlsanityData to 7 Oct 2026

Demystifying Prompts in Language Models via Perplexity Estimation

arXiv 2212.04037 ↗First posted 8 Dec 2022cs.CLcited by 215

Not on the map. Why?

Abstract (arXiv)

Language models can be prompted to perform a wide variety of zero- and few-shot learning problems. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens or how to pick the best prompts. In this work, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task. As a result, we devise a method for creating prompts: (1) automatically extend a small seed set of manually written prompts by paraphrasing using GPT3 and backtranslation and (2) choose the lowest perplexity prompts to get significant gains in performance.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

24 of 47 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  12. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
12 more
  1. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  2. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  3. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  4. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  5. 2021How Many Data Points is a Prompt Worth?cited by 197
  6. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  7. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  8. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  9. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  10. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  11. 2017Learning Paraphrastic Sentence Embeddings from Back-Translated Bitextcited by 27
  12. 2018Multi-Source Social Feedback of Online News Feedscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 49; 2024: 75; 2025: 69; 2026: 19.

215 papers.

Sort:

By area:

  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  4. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  5. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  7. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  8. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  9. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  10. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  11. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  12. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
38 more
  1. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  2. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  3. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  4. 2023In-context Example Selection with Influencescited by 60
  5. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  6. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  7. 2024Compression Represents Intelligence Linearlycited by 56
  8. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  9. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  10. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  11. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  12. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  13. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  14. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 43
  15. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
  16. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  17. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  18. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  19. 2024On the Worst Prompt Performance of Large Language Modelscited by 36
  20. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  21. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  22. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  23. 2023Prompt Optimization via Adversarial In-Context Learningcited by 33
  24. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  25. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  26. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  27. 2024Smoothie: Label Free Language Model Routingcited by 31
  28. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  29. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Studycited by 29
  30. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  31. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
  32. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  33. 2023Pushing the Limits of ChatGPT on NLP Taskscited by 27
  34. 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
  35. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  36. 2024Do LLMs "know" internally when they follow instructions?cited by 25
  37. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  38. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25

50 most cited of 215 All 215 (JSON)