# Criteria-Based LLM Relevance Judgments

arXiv: https://arxiv.org/abs/2507.09488 · HTML: https://mlsanity.com/paper/2507.09488/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2025.

## Built on by (most cited first, 4 of 4)

- [Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers](https://mlsanity.com/paper/2601.03211/) (2026)
- [Supporting Humans in Evaluating AI Summaries of Legal Depositions](https://mlsanity.com/paper/2601.15182/) (2026)
- [NBQ: Next-Best-Question for Dynamic Profiling](https://mlsanity.com/paper/2606.00809/) (2026)
- [Sharding Prevents LLM Oversight Failures and Adversarial Exploitation](https://mlsanity.com/paper/2608.06422/) (2026)

## Abstract (arXiv)

Relevance judgments are crucial for evaluating information retrieval systems, but traditional human-annotated labels are time-consuming and expensive. As a result, many researchers turn to automatic alternatives to accelerate method development. Among these, Large Language Models (LLMs) provide a scalable solution by generating relevance labels directly through prompting. However, prompting an LLM for a relevance label without constraints often results in not only incorrect predictions but also outputs that are difficult for humans to interpret. We propose the Multi-Criteria framework for LLM-based relevance judgments, decomposing the notion of relevance into multiple criteria--such as exactness, coverage, topicality, and contextual fit--to improve the robustness and interpretability of retrieval evaluations compared to direct grading methods. We validate this approach on three datasets: the TREC Deep Learning tracks from 2019 and 2020, as well as LLMJudge (based on TREC DL 2023). Our results demonstrate that Multi-Criteria judgments enhance the system ranking/leaderboard performance. Moreover, we highlight the strengths and limitations of this approach relative to direct grading approaches, offering insights that can guide the development of future automatic evaluation frameworks in information retrieval.