# UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval

arXiv: https://arxiv.org/abs/2604.15827 · HTML: https://mlsanity.com/paper/2604.15827/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks](https://mlsanity.com/paper/2410.20266/) (2024)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools](https://mlsanity.com/paper/2307.15770/) (2023)
- [Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals](https://mlsanity.com/paper/2502.16101/) (2025)
- [ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists](https://mlsanity.com/paper/2506.01241/) (2025)
- [Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective](https://mlsanity.com/paper/2411.14572/) (2024)
- [Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition](https://mlsanity.com/paper/2410.05603/) (2024)
- [Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives](https://mlsanity.com/paper/2408.06904/) (2024)
- [ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures](https://mlsanity.com/paper/2406.09818/) (2024)
- [Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.05220/) (2025)

## Built on by (most cited first, 1 of 1)

- [Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG](https://mlsanity.com/paper/2607.17742/) (2026)

## Abstract (arXiv)

Conventional information retrieval is concerned with identifying the relevance of texts for a given query. Yet, the conventional definition of relevance is dominated by aspects of similarity in texts, leaving unobserved whether the text is truly useful for addressing the query. For instance, when answering whether Paris is larger than Berlin, texts about Paris being in France are relevant (lexical/semantic similarity), but not useful. In this paper, we introduce UsefulBench, a domain-specific dataset curated by three professional analysts labeling whether a text is connected to a query (relevance) or holds practical value in responding to it (usefulness). We show that classic similarity-based information retrieval aligns more strongly with relevance. While LLM-based systems can counteract this bias, we find that domain-specific problems require a high degree of expertise, which current LLMs do not fully incorporate. We explore approaches to (partially) overcome this challenge. However, UsefulBench presents a dataset challenge for targeted information retrieval systems.