mlsanityData to 14 Oct 2026

When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance

arXiv 2606.26062 ↗First posted 24 Jun 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Can a statistically significant, large-effect-size finding in computational social science be entirely an artifact of the measurement instrument? We present a case where the answer appears to be yes. Analyzing 85 interviews across four public intellectuals (2016--2026), we find a robust negative-affect/emphatic-certainty lexical co-occurrence pattern under keyword-based scoring (r = 0.72--0.93, p < 0.01 for all four speakers). Replacing keyword counting with LLM-based zero-shot semantic classification on the complete diarized corpus (32,625 sentences) dramatically reduces this correlation: Dalio's r = 0.851 drops to r = 0.206, with two speakers showing negative r(neg, emphatic) and one showing null. In contrast, the LLM reveals a strong negative-hedging coupling across speakers -- Rogoff's r(neg, hedged) = 0.875 (p = 0.001) and Zeihan's r(neg, hedged) = 0.722 (p = 0.008) -- consistent with the conventional expectation that pessimistic discourse attracts hedging, not certainty. Sentence-level error analysis traces this discrepancy to three structural failure modes in keyword lexicons -- syntactic blindness, polysemy blindness, and categorical absence -- illustrated through cases where keyword counting inverts semantic meaning (e.g., ''never absolutely totally confident'' scored as high-certainty). We argue that keyword lexicons measure a universal lexical co-occurrence tendency -- negative discourse naturally attracts emphatic vocabulary -- that is orthogonal to, and can systematically invert, rhetorical stance. Treating keyword counts as measurements of epistemic certainty is a category error: a finding that appears to be about a speaker's psychology may be entirely about the counting of words.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 7 of its 24 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

7 of 24 references.

References · 7

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  5. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  6. 2020How Context Affects Language Models' Factual Predictionscited by 181
  7. 2024Financial Statement Analysis with Large Language Modelscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.