# The simulation of judgment in LLMs

arXiv: https://arxiv.org/abs/2502.04426 · HTML: https://mlsanity.com/paper/2502.04426/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 64 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023)
- [The COVID-19 Social Media Infodemic](https://mlsanity.com/paper/2003.05004/) (2020)
- [Large Language Models Can Be Used to Estimate the Latent Positions of Politicians](https://mlsanity.com/paper/2303.12057/) (2023)
- [The Perils & Promises of Fact-checking with Large Language Models](https://mlsanity.com/paper/2310.13549/) (2023)
- [Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina](https://mlsanity.com/paper/2410.19599/) (2024)
- [Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models](https://mlsanity.com/paper/2304.00228/) (2023)
- [Unmasking Conversational Bias in AI Multiagent Systems](https://mlsanity.com/paper/2501.14844/) (2025)
- [Studying and improving reasoning in humans and machines](https://mlsanity.com/paper/2309.12485/) (2023)
- [LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains](https://mlsanity.com/paper/2407.14344/) (2024)

## Built on by (most cited first, 22 of 22)

- [Epistemological Fault Lines Between Human and Artificial Intelligence](https://mlsanity.com/paper/2512.19466/) (2025)
- [Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity](https://mlsanity.com/paper/2507.00657/) (2025)
- [Social physics in the age of artificial intelligence](https://mlsanity.com/paper/2603.16900/) (2026)
- [Beyond "Hallucinations": A Framework for Stable Human-AI Reasoning](https://mlsanity.com/paper/2510.14665/) (2025)
- [A closer look at how large language models trust humans: patterns and biases](https://mlsanity.com/paper/2504.15801/) (2025)
- [The Statistical Signature of LLMs](https://mlsanity.com/paper/2602.18152/) (2026)
- [AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized](https://mlsanity.com/paper/2607.13562/) (2026)
- [The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse](https://mlsanity.com/paper/2608.10186/) (2026)
- [Cognitive networks highlight differences and similarities in the STEM mindsets of human and LLM-simulated trainees, experts and academics](https://mlsanity.com/paper/2502.19529/) (2025)
- [Generative AI and the future of scientometrics: current topics and future questions](https://mlsanity.com/paper/2507.00783/) (2025)
- [Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Models](https://mlsanity.com/paper/2604.13706/) (2026)
- [Quantifying Systemic Vulnerability in the Foundation Model Industry](https://mlsanity.com/paper/2510.23421/) (2025)
- [Do LLM-Driven Agents Exhibit Engagement Mechanisms? Controlled Tests of Information Load, Descriptive Norms, and Popularity Cues](https://mlsanity.com/paper/2603.20911/) (2026)
- [Task Ecologies and the Evolution of World-Tracking Representations in Large Language Models](https://mlsanity.com/paper/2604.05469/) (2026)
- [Large language models eroding science understanding: an empirical study of malignment](https://mlsanity.com/paper/2604.25639/) (2026)
- [Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Grid](https://mlsanity.com/paper/2605.01359/) (2026)
- [Are LLMs More Skeptical of Entertainment News?](https://mlsanity.com/paper/2605.01727/) (2026)
- [LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems](https://mlsanity.com/paper/2606.02883/) (2026)
- [Collective cooperation without individual fidelity in LLM agents](https://mlsanity.com/paper/2606.30454/) (2026)
- [What is Left for Us? Second Scholarship Against the Degradation of Research by AI](https://mlsanity.com/paper/2607.04049/) (2026)
- [Beyond Epistemia: Epistemic Schizologia and Large Language Models as Techno-Semiotic Machines](https://mlsanity.com/paper/2607.25620/) (2026)
- [Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content](https://mlsanity.com/paper/2609.30563/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly embedded in evaluative processes, from information filtering to assessing and addressing knowledge gaps through explanation and credibility judgments. This raises the need to examine how such evaluations are built, what assumptions they rely on, and how their strategies diverge from those of humans. We benchmark six LLMs against expert ratings--NewsGuard and Media Bias/Fact Check--and against human judgments collected through a controlled experiment. We use news domains purely as a controlled benchmark for evaluative tasks, focusing on the underlying mechanisms rather than on news classification per se. To enable direct comparison, we implement a structured agentic framework in which both models and nonexpert participants follow the same evaluation procedure: selecting criteria, retrieving content, and producing justifications. Despite output alignment, our findings show consistent differences in the observable criteria guiding model evaluations, suggesting that lexical associations and statistical priors could influence evaluations in ways that differ from contextual reasoning. This reliance is associated with systematic effects: political asymmetries and a tendency to confuse linguistic form with epistemic reliability--a dynamic we term epistemia, the illusion of knowledge that emerges when surface plausibility replaces verification. Indeed, delegating judgment to such systems may affect the heuristics underlying evaluative processes, suggesting a shift from normative reasoning toward pattern-based approximation and raising open questions about the role of LLMs in evaluative processes.