Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
Not on the map. Why?
Abstract (arXiv)
Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 61 references that are in the map. First posted on arXiv 29 Nov 2025.
Builds on
21 of 61 references.
References · 21
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
- 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
- 2023Large language models can accurately predict searcher preferencescited by 131
- 2024Cognitive Bias in Decision-Making with LLMscited by 106
- 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
- 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
- 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
- 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
- 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
- 2024LLMs Simulate Big Five Personality Traits: Further Evidencecited by 22
- 2025Overview of the TREC 2022 deep learning trackcited by 22
9 more
- 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
- 2025Overview of the TREC 2021 deep learning trackcited by 18
- 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
- 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
- 2023Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentationcited by 11
- 2025Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Rerankingcited by 10
- 2025Exploring the Potential of Large Language Models to Simulate Personalitycited by 7
- 2024Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerabilitycited by 6
- 2025Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.