# Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models

arXiv: https://arxiv.org/abs/2403.14859 · HTML: https://mlsanity.com/paper/2403.14859/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions](https://mlsanity.com/paper/2207.14251/) (2022)
- [Language models align with human judgments on key grammatical constructions](https://mlsanity.com/paper/2402.01676/) (2024)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [When classifying grammatical role, BERT doesn't care about word order... except when it matters](https://mlsanity.com/paper/2203.06204/) (2022)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [How well does surprisal explain N400 amplitude under different experimental conditions?](https://mlsanity.com/paper/2010.04844/) (2020)
- [Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge](https://mlsanity.com/paper/2107.10922/) (2021)
- [The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation](https://mlsanity.com/paper/2109.07848/) (2021)
- [When Language Models Fall in Love: Animacy Processing in Transformer Language Models](https://mlsanity.com/paper/2310.15004/) (2023)
- [A Structured Distributional Model of Sentence Meaning and Processing](https://mlsanity.com/paper/1906.07280/) (2019)
- [Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently](https://mlsanity.com/paper/2401.06640/) (2024)

## Built on by (most cited first, 24 of 24)

- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks](https://mlsanity.com/paper/2506.13351/) (2025)
- [The "LLM World of Words" English free association norms generated by large language models](https://mlsanity.com/paper/2412.01330/) (2024)
- [Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition](https://mlsanity.com/paper/2503.10037/) (2025)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models](https://mlsanity.com/paper/2502.15487/) (2025)
- [Human Psychometric Questionnaires Mischaracterize LLM Behavior](https://mlsanity.com/paper/2509.10078/) (2025)
- [MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG](https://mlsanity.com/paper/2604.24564/) (2026)
- [The quasi-semantic competence of LLMs: a case study on the part-whole relation](https://mlsanity.com/paper/2504.02395/) (2025)
- [Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLM](https://mlsanity.com/paper/2602.18693/) (2026)
- [Predicting Sentence Acceptability Judgments in Multimodal Contexts](https://mlsanity.com/paper/2602.20918/) (2026)
- [Humans and LLMs Diverge on Probabilistic Inferences](https://mlsanity.com/paper/2602.23546/) (2026)
- [Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training](https://mlsanity.com/paper/2605.02241/) (2026)
- [Implicit Representations of Grammaticality in Language Models](https://mlsanity.com/paper/2605.05197/) (2026)
- [Measuring Form and Function in Language Models](https://mlsanity.com/paper/2605.28616/) (2026)
- [Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation](https://mlsanity.com/paper/2410.15173/) (2024)
- [From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models](https://mlsanity.com/paper/2508.18253/) (2025)
- [PerHalluEval: Persian Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2509.21104/) (2025)
- [Contextualising (Im)plausible Events Triggers Figurative Language](https://mlsanity.com/paper/2604.07885/) (2026)
- [Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from Demonstratives](https://mlsanity.com/paper/2604.25423/) (2026)
- [Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMs](https://mlsanity.com/paper/2605.08149/) (2026)
- [CoCoA: Context-Conditional Cultural Alignment for Large Language Models](https://mlsanity.com/paper/2608.29492/) (2026)
- [Likelihood Ranking doesn't Scale Like Prompting in LLMs](https://mlsanity.com/paper/2609.29390/) (2026)

## Abstract (arXiv)

Semantic plausibility (e.g. knowing that "the actor won the award" is more likely than "the actor won the battle") serves as an effective proxy for general world knowledge. Language models (LMs) capture vast amounts of world knowledge by learning distributional patterns in text, accessible via log probabilities (LogProbs) they assign to plausible vs. implausible outputs. The new generation of instruction-tuned LMs can now also provide explicit estimates of plausibility via prompting. Here, we evaluate the effectiveness of LogProbs and basic prompting to measure semantic plausibility, both in single-sentence minimal pairs (Experiment 1) and short context-dependent scenarios (Experiment 2). We find that (i) in both base and instruction-tuned LMs, LogProbs offers a more reliable measure of semantic plausibility than direct zero-shot prompting, which yields inconsistent and often poor results; (ii) instruction-tuning generally does not alter the sensitivity of LogProbs to semantic plausibility (although sometimes decreases it); (iii) across models, context mostly modulates LogProbs in expected ways, as measured by three novel metrics of context-sensitive plausibility and their match to explicit human plausibility judgments. We conclude that, even in the era of prompt-based evaluations, LogProbs constitute a useful metric of semantic plausibility, both in base and instruction-tuned LMs.