# When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search

arXiv: https://arxiv.org/abs/2507.02139 · HTML: https://mlsanity.com/paper/2507.02139/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 7 of 7)

- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [A Comparison of Methods for Evaluating Generative IR](https://mlsanity.com/paper/2404.04044/) (2024)
- [Quati: A Brazilian Portuguese Information Retrieval Dataset from Native Speakers](https://mlsanity.com/paper/2404.06976/) (2024)
- [Exploring Large Language Models for Relevance Judgments in Tetun](https://mlsanity.com/paper/2406.07299/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines, especially in domains lacking human-labeled data. However, different models often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval. This study examines labeling disagreement between two open-weight LLMs, LLaMA and Qwen, on a corpus of scholarly abstracts related to Sustainable Development Goals (SDGs) 1, 3, and 7. We isolate disagreement subsets and examine their lexical properties, rank-order behavior, and classification predictability. Our results show that model disagreement is systematic, not random: disagreement cases exhibit consistent lexical patterns, produce divergent top-ranked outputs under shared scoring functions, and are distinguishable with AUCs above 0.74 using simple classifiers. These findings suggest that LLM-based filtering introduces structured variability in document retrieval, even under controlled prompting and shared ranking logic. We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.