mlsanityData to 7 Oct 2026

Prompting is not a substitute for probability measurements in large language models

arXiv 2305.13264 ↗First posted 22 May 2023cs.CL cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

Prompting is now a dominant method for evaluating the linguistic knowledge of large language models (LLMs). While other methods directly read out models' probability distributions over strings, prompting requires models to access this internal information by processing linguistic input, thereby implicitly testing a new type of emergent ability: metalinguistic judgment. In this study, we compare metalinguistic prompting and direct probability measurements as ways of measuring models' linguistic knowledge. Broadly, we find that LLMs' metalinguistic judgments are inferior to quantities directly derived from representations. Furthermore, consistency gets worse as the prompt query diverges from direct measurements of next-word probabilities. Our findings suggest that negative results relying on metalinguistic prompts cannot be taken as conclusive evidence that an LLM lacks a particular linguistic generalization. Our results also highlight the value that is lost with the move to closed APIs where access to probability distributions is limited.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

27 of 51 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
15 more
  1. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  3. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  4. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  5. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  6. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  7. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  8. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  9. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  10. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  11. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  12. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  13. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  14. 2023Evaluating statistical language models as pragmatic reasonerscited by 14
  15. 2021Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 17; 2026: 7.

43 papers.

Sort:

By area:

  1. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  2. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  3. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  4. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  5. 2024Language models align with human judgments on key grammatical constructionscited by 49
  6. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  7. 2024On the Challenges and Opportunities in Generative AIcited by 31
  8. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  9. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  10. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  11. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  12. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
31 more
  1. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  2. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  3. 2024Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extractioncited by 6
  4. 2025Agree to Disagree? A Meta-Evaluation of LLM Misgenderingcited by 6
  5. 2025The Consistency Hypothesis in Uncertainty Quantification for Large Language Modelscited by 6
  6. 2023Large language models and (non-)linguistic recursioncited by 5
  7. 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
  8. 2025Aligned Probing: Relating Toxic Behavior and Model Internalscited by 5
  9. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  10. 2024RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domaincited by 4
  11. 2024The "LLM World of Words" English free association norms generated by large language modelscited by 4
  12. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  13. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  14. 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratingscited by 3
  15. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  16. 2023Attention, not scale, drives human-AI alignment in multimodal language predictioncited by 1
  17. 2025MIAT: Maneuver-Intention-Aware Transformer for Spatio-Temporal Trajectory Predictioncited by 1
  18. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobscited by 1
  19. 2025For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependenciescited by 1
  20. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Modelscited by 1
  21. 2023Conceptual Engineering Using Large Language Modelscited by 0
  22. 2024Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Modelscited by 0
  23. 2024MacBehaviour: An R package for behavioural experimentation on large language modelscited by 0
  24. 2025Do Large Language Models know who did what to whom?cited by 0
  25. 2025Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attractioncited by 0
  26. 2025Can GPT replace human raters? Validity and reliability of machine-generated norms for metaphorscited by 0
  27. 2026To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Timescited by 0
  28. 2026Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoningcited by 0
  29. 2026Linear representations of grammaticality in neural language modelscited by 0
  30. 2026Enabling Creative Exploration for Vibe Design Agentscited by 0
  31. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0