# An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs

arXiv: https://arxiv.org/abs/2406.11290 · HTML: https://mlsanity.com/paper/2406.11290/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 55 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [LLatrieval: LLM-Verified Retrieval for Verifiable Generation](https://mlsanity.com/paper/2311.07838/) (2023)
- [ANTIQUE: A Non-Factoid Question Answering Benchmark](https://mlsanity.com/paper/1905.08957/) (2019)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval](https://mlsanity.com/paper/1805.03797/) (2018)
- [From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification](https://mlsanity.com/paper/2310.11675/) (2023)
- [Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness](https://mlsanity.com/paper/2405.02714/) (2024)
- [Iterative Relevance Feedback for Answer Passage Retrieval with Passage-level Semantic Match](https://mlsanity.com/paper/1812.08870/) (2018)
- [Characterizing Information Seeking Processes with Multiple Physiological Signals](https://mlsanity.com/paper/2405.00322/) (2024)

## Built on by (most cited first, 3 of 3)

- [Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.05220/) (2025)
- [Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.19102/) (2025)
- [AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.25092/) (2026)

## Abstract (arXiv)

Relevance and utility are two frequently used measures to evaluate the effectiveness of an information retrieval (IR) system. Relevance emphasizes the aboutness of a result to a query, while utility refers to the result's usefulness or value to an information seeker. In retrieval-augmented generation (RAG), high-utility results should be prioritized to feed to LLMs due to their limited input bandwidth. Re-examining RAG's three core components-relevance ranking derived from retrieval models, utility judgments, and answer generation-aligns with Schutz's philosophical system of relevances, which encompasses three types of relevance representing different levels of human cognition that enhance each other. These three RAG components also reflect three cognitive levels for LLMs in question-answering. Therefore, we propose an Iterative utiliTy judgmEnt fraMework (ITEM) to promote each step in RAG. We conducted extensive experiments on retrieval (TREC DL, WebAP), utility judgment task (GTI-NQ), and factoid question-answering (NQ) datasets. Experimental results demonstrate improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.