# Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment arXiv: https://arxiv.org/abs/2601.01862 ยท HTML: https://mlsanity.com/paper/2601.01862/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 82 references that are in the map. - First posted on arXiv 5 Jan 2026. ## Builds on (most cited first, 30 of 34) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023) - [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020) - [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020) - [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024) - [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023) - [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021) - [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023) - [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023) - [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023) - [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024) - [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025) - [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023) - [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024) - [LLM-Powered User Simulator for Recommender System](https://mlsanity.com/paper/2412.16984/) (2024) - [One-Shot Labeling for Automatic Relevance Estimation](https://mlsanity.com/paper/2302.11266/) (2023) - [Cycles of Thought: Measuring LLM Confidence through Stable Explanations](https://mlsanity.com/paper/2406.03441/) (2024) - [LLM-based relevance assessment still can't replace human relevance assessment](https://mlsanity.com/paper/2412.17156/) (2024) - [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024) - [LLMs Simulate Big Five Personality Traits: Further Evidence](https://mlsanity.com/paper/2402.01765/) (2024) - [Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience](https://mlsanity.com/paper/2404.10315/) (2024) - [AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment](https://mlsanity.com/paper/2409.16022/) (2024) - [Overview of the TREC 2021 deep learning track](https://mlsanity.com/paper/2507.08191/) (2025) - [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025) - [A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment](https://mlsanity.com/paper/2504.12408/) (2025) - [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025) - [LLMs can be Fooled into Labelling a Document as Relevant (best caf'e near me; this paper is perfectly relevant)](https://mlsanity.com/paper/2501.17969/) (2025) - [Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking](https://mlsanity.com/paper/2509.11353/) (2025) - [Exploring the Potential of Large Language Models to Simulate Personality](https://mlsanity.com/paper/2502.08265/) (2025) ## Built on by (most cited first, 1 of 1) - [Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation](https://mlsanity.com/paper/2608.10385/) (2026) ## Abstract (arXiv) Recent studies have shown that prompting can enable large language models (LLMs) to simulate specific personality traits and produce behaviors that align with those traits. However, there is limited understanding of how these simulated personalities influence critical web search decisions, specifically relevance assessment. Moreover, few studies have examined how simulated personalities impact confidence calibration, specifically the tendencies toward overconfidence or underconfidence. This gap exists even though psychological literature suggests these biases are trait-specific, often linking high extraversion to overconfidence and high neuroticism to underconfidence. To address this gap, we conducted a comprehensive study evaluating multiple LLMs, including commercial models and open-source models, prompted to simulate Big Five personality traits. We tested these models across three test collections (TREC DL 2019, TREC DL 2020, and LLMJudge), collecting two key outputs for each query-document pair: a relevance judgment and a self-reported confidence score. The findings show that personalities such as low agreeableness consistently align more closely with human labels than the unprompted condition. Additionally, low conscientiousness performs well in balancing the suppression of both overconfidence and underconfidence. We also observe that relevance scores and confidence distributions vary systematically across different personalities. Based on the above findings, we incorporate personality-conditioned scores and confidence as features in a random forest classifier. This approach achieves performance that surpasses the best single-personality condition on a new dataset (TREC DL 2021), even with limited training data. These findings highlight that personality-derived confidence offers a complementary predictive signal, paving the way for more reliable and human-aligned LLM evaluators.