Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison. We study persona conditioning as a diagnostic mechanism for exposing LLM assessor sensitivity. Using task-oriented personas drawn from two complementary sources (PersonaHub and NVIDIA Nemotron-Personas-USA), we instantiate five assessor roles emphasizing intent interpretation, domain expertise, contrastive judgment, evidence verification, and global search-quality assessment, compared with a standard UMBRELA baseline. Across six LLM backbones on TREC DL20 and RAG24, our analyses reveal structured rather than uniform assessor sensitivity. Judgments usually remain close to the baseline while shifting assessment strictness, evidential threshold, or interpretation emphasis rather than producing widespread relevance reversals. At the system level, high-capacity models preserve system-ranking agreement, while smaller models amplify persona-induced instability. Local rank-displacement analysis shows sensitivity concentrates on particular retrieval systems and system types, especially neural ranking/reranking systems on DL20 and RAG-oriented pipelines on RAG24. Persona source matters less than assessor role and model capacity. These findings position persona-conditioned judging as a controlled sensitivity probe for stress-testing LLM-based IR evaluation pipelines and identifying systems whose evaluation outcomes are sensitive to assessor framing.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 63 references that are in the map. First posted on arXiv 11 Aug 2026.
Builds on
35 of 63 references.
References · 35
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
- 2021Overview of the TREC 2020 deep learning trackcited by 196
- 2023Large language models can accurately predict searcher preferencescited by 131
- 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
- 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
- 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 32
23 more
- 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
- 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
- 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
- 2024LLMs Simulate Big Five Personality Traits: Further Evidencecited by 22
- 2024Personas with Attitudes: Controlling LLMs for Diverse Data Annotationcited by 21
- 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
- 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
- 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
- 2024Unveiling the Lexical Sensitivity of LLMs: Combinatorial Optimization for Prompt Enhancementcited by 13
- 2024LLMs Can Patch Up Missing Relevance Judgments in Evaluationcited by 12
- 2025LLMs can be Fooled into Labelling a Document as Relevant (best caf'e near me; this paper is perfectly relevant)cited by 10
- 2025Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Rerankingcited by 10
- 2025The Effects of Demographic Instructions on LLM Personascited by 5
- 2025Political Ideology Shifts in Large Language Modelscited by 5
- 2025CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Modelscited by 5
- 2024SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMscited by 3
- 2025Does UMBRELA Work on Other LLMs?cited by 2
- 2025How role-play shapes relevance judgment in zero-shot LLM rankerscited by 1
- 2025Ideology-Based LLMs for Content Moderationcited by 1
- 2026Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysiscited by 1
- 2026Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessmentcited by 1
- 2026When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessmentcited by 1
- 2026Formalized Information Needs Improve Large-Language-Model Relevance Judgmentscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026LLMs as Assessors: Right for the Right Reason?
- 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
- 2025Exploring the Impact of Occupational Personas on Domain-Specific QA
- 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
- 2026The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs
- 2024PersonaGym: Evaluating Persona Agents and LLMs
- 2026Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring
- 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look