mlsanityData to 7 Oct 2026

An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels

arXiv 2203.11364 ↗First posted 21 Mar 2022cs.CL cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models derive substantial linguistic and factual knowledge from the massive corpora on which they are trained, and prompt engineering seeks to align these models to specific tasks. Unfortunately, existing prompt engineering methods require significant amounts of labeled data, access to model parameters, or both. We introduce a new method for selecting prompt templates textitwithout labeled examples and textitwithout direct access to the model. Specifically, over a set of candidate templates, we choose the template that maximizes the mutual information between the input and the corresponding model output. Across 8 datasets representing 7 distinct NLP tasks, we show that when a template has high mutual information, it also has high accuracy on the task. On the largest model, selecting prompts with our method gets 90% of the way from the average prompt accuracy to the best prompt accuracy and requires no ground truth labels.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 27 of its 64 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

27 of 64 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
15 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2021GPT Understands, Toocited by 715
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  5. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  6. 2021True Few-Shot Learning with Language Modelscited by 343
  7. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  8. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  9. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  10. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  11. 2019Inducing Relational Knowledge from BERTcited by 75
  12. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  13. 2021Early-stopped neural networks are consistentcited by 34
  14. 2018Word Sense Induction with Neural biLM and Symmetric Patternscited by 18
  15. 2018Mutual Information Maximization for Simple and Accurate Part-Of-Speech Inductioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 23; 2024: 32; 2025: 20; 2026: 9.

88 papers.

Sort:

By area:

  1. 2022A Survey on In-context Learningcited by 717
  2. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  5. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  6. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  7. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  8. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  9. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  10. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  11. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
  12. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
38 more
  1. 2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
  2. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  3. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  4. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  5. 2024Smoothie: Label Free Language Model Routingcited by 31
  6. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  7. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  8. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  9. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  10. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  11. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Modelscited by 18
  12. 2024Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classificationcited by 16
  13. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  14. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  15. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  16. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  17. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  18. 2024Implicit In-context Learningcited by 11
  19. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  20. 2024Large Language Models Can Better Understand Knowledge Graphs Than We Thoughtcited by 10
  21. 2024Laurel: Unblocking Automated Verification with Large Language Modelscited by 10
  22. 2025Meaning Is Not A Metric: Using LLMs to make cultural context legible at scalecited by 10
  23. 2024Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Processcited by 9
  24. 2023LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extractioncited by 8
  25. 2024Revisiting In-Context Learning with Long Context Language Modelscited by 7
  26. 2023Dynamic Demonstrations Controller for In-Context Learningcited by 6
  27. 2023On Meta-Promptingcited by 6
  28. 2024Enhancing In-Context Learning via Implicit Demonstration Augmentationcited by 6
  29. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  30. 2025CAPO: Cost-Aware Prompt Optimizationcited by 6
  31. 2023Construction contract risk identification based on knowledge-augmented language modelcited by 5
  32. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5
  33. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  34. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  35. 2024SADL: An Effective In-Context Learning Method for Compositional Visual QAcited by 5
  36. 2024CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Modelscited by 5
  37. 2024Curriculum Demonstration Selection for In-Context Learningcited by 5
  38. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5

50 most cited of 88 All 88 (JSON)