# Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing

arXiv: https://arxiv.org/abs/2512.00390 · HTML: https://mlsanity.com/paper/2512.00390/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 29 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [LLM-based relevance assessment still can't replace human relevance assessment](https://mlsanity.com/paper/2412.17156/) (2024)
- [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024)
- [LLMs Simulate Big Five Personality Traits: Further Evidence](https://mlsanity.com/paper/2402.01765/) (2024)
- [Overview of the TREC 2022 deep learning track](https://mlsanity.com/paper/2507.10865/) (2025)
- [AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment](https://mlsanity.com/paper/2409.16022/) (2024)
- [Overview of the TREC 2021 deep learning track](https://mlsanity.com/paper/2507.08191/) (2025)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation](https://mlsanity.com/paper/2310.16738/) (2023)
- [Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking](https://mlsanity.com/paper/2509.11353/) (2025)
- [Exploring the Potential of Large Language Models to Simulate Personality](https://mlsanity.com/paper/2502.08265/) (2025)
- [Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability](https://mlsanity.com/paper/2403.18462/) (2024)
- [Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs](https://mlsanity.com/paper/2506.17782/) (2025)

## Built on by (most cited first, 2 of 2)

- [Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment](https://mlsanity.com/paper/2601.01862/) (2026)
- [Experiences Build Characters: The Linguistic Origins and Functional Impact of LLM Personality](https://mlsanity.com/paper/2603.06088/) (2026)

## Abstract (arXiv)

Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices.