When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance
Not on the map. Why?
Abstract (arXiv)
Can a statistically significant, large-effect-size finding in computational social science be entirely an artifact of the measurement instrument? We present a case where the answer appears to be yes. Analyzing 85 interviews across four public intellectuals (2016--2026), we find a robust negative-affect/emphatic-certainty lexical co-occurrence pattern under keyword-based scoring (r = 0.72--0.93, p < 0.01 for all four speakers). Replacing keyword counting with LLM-based zero-shot semantic classification on the complete diarized corpus (32,625 sentences) dramatically reduces this correlation: Dalio's r = 0.851 drops to r = 0.206, with two speakers showing negative r(neg, emphatic) and one showing null. In contrast, the LLM reveals a strong negative-hedging coupling across speakers -- Rogoff's r(neg, hedged) = 0.875 (p = 0.001) and Zeihan's r(neg, hedged) = 0.722 (p = 0.008) -- consistent with the conventional expectation that pessimistic discourse attracts hedging, not certainty. Sentence-level error analysis traces this discrepancy to three structural failure modes in keyword lexicons -- syntactic blindness, polysemy blindness, and categorical absence -- illustrated through cases where keyword counting inverts semantic meaning (e.g., ''never absolutely totally confident'' scored as high-certainty). We argue that keyword lexicons measure a universal lexical co-occurrence tendency -- negative discourse naturally attracts emphatic vocabulary -- that is orthogonal to, and can systematically invert, rhetorical stance. Treating keyword counts as measurements of epistemic certainty is a category error: a finding that appears to be about a speaker's psychology may be entirely about the counting of words.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 7 of its 24 references that are in the map. First posted on arXiv 24 Jun 2026.
Builds on
7 of 24 references.
References · 7
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2023Can Large Language Models Transform Computational Social Science?cited by 250
- 2020How Context Affects Language Models' Factual Predictionscited by 181
- 2024Financial Statement Analysis with Large Language Modelscited by 31
Built on by
Citing papers per year: 2026: 2.
2 papers.