# A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment

arXiv: https://arxiv.org/abs/2504.12408 · HTML: https://mlsanity.com/paper/2504.12408/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels](https://mlsanity.com/paper/2310.14122/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [The language of prompting: What linguistic properties make a prompt successful?](https://mlsanity.com/paper/2311.01967/) (2023)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [How are Prompts Different in Terms of Sensitivity?](https://mlsanity.com/paper/2311.07230/) (2023)
- [One-Shot Labeling for Automatic Relevance Estimation](https://mlsanity.com/paper/2302.11266/) (2023)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look](https://mlsanity.com/paper/2411.08275/) (2024)
- [Assessing and Verifying Task Utility in LLM-Powered Applications](https://mlsanity.com/paper/2405.02178/) (2024)
- [Human Preferences as Dueling Bandits](https://mlsanity.com/paper/2204.10362/) (2022)
- [A Comparison of Methods for Evaluating Generative IR](https://mlsanity.com/paper/2404.04044/) (2024)
- [Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024](https://mlsanity.com/paper/2408.05388/) (2024)
- [Assessing top-k preferences](https://mlsanity.com/paper/2007.11682/) (2020)

## Built on by (most cited first, 15 of 15)

- [Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs](https://mlsanity.com/paper/2505.16967/) (2025)
- [Silenced Biases: The Dark Side LLMs Learned to Refuse](https://mlsanity.com/paper/2511.03369/) (2025)
- [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026)
- [Eval4Sim: An Evaluation Framework for Persona Simulation](https://mlsanity.com/paper/2603.02876/) (2026)
- [From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences](https://mlsanity.com/paper/2604.10332/) (2026)
- [Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework](https://mlsanity.com/paper/2606.08867/) (2026)
- [Relative Bias: A Comparative Framework for Quantifying Bias in LLMs](https://mlsanity.com/paper/2505.17131/) (2025)
- [Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment](https://mlsanity.com/paper/2601.01862/) (2026)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search](https://mlsanity.com/paper/2609.29803/) (2026)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [The EpisTwin: A Knowledge Graph-Grounded Neuro-Symbolic Architecture for Personal AI](https://mlsanity.com/paper/2603.06290/) (2026)
- [PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses](https://mlsanity.com/paper/2603.10477/) (2026)
- [A Formal Framework for Uncertainty Analysis of Text Generation with Large Language Models](https://mlsanity.com/paper/2603.26363/) (2026)
- [Embedding Surgery: Localized Updates for Adaptive Ranking Correction in Dense Retrieval](https://mlsanity.com/paper/2609.05110/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks~ -- ~binary, graded, and pairwise~ -- ~yielding 90 prompts in total. After filtering out unusable prompts from three humans and three LLMs, we employed the remaining 72 prompts with three different LLMs as judges to label document/query pairs from two TREC Deep Learning Datasets (2020 and 2021). We compare LLM-generated labels with TREC official human labels using Cohen's kappa and pairwise agreement measures. In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We also compare human- and LLM-generated prompts with the standard UMBRELA prompt used for relevance assessment by Bing and TREC 2024 Retrieval Augmented Generation (RAG) Track. To support future research in LLM-based evaluation, we release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.