UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
Not on the map. Why?
Abstract (arXiv)
Conventional information retrieval is concerned with identifying the relevance of texts for a given query. Yet, the conventional definition of relevance is dominated by aspects of similarity in texts, leaving unobserved whether the text is truly useful for addressing the query. For instance, when answering whether Paris is larger than Berlin, texts about Paris being in France are relevant (lexical/semantic similarity), but not useful. In this paper, we introduce UsefulBench, a domain-specific dataset curated by three professional analysts labeling whether a text is connected to a query (relevance) or holds practical value in responding to it (usefulness). We show that classic similarity-based information retrieval aligns more strongly with relevance. While LLM-based systems can counteract this bias, we find that domain-specific problems require a high degree of expertise, which current LLMs do not fully incorporate. We explore approaches to (partially) overcome this challenge. However, UsefulBench presents a dataset challenge for targeted information retrieval systems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 17 Apr 2026.
Builds on
22 of 37 references.
References · 22
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
- 2023Evaluating Verifiability in Generative Search Enginescited by 297
- 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
- 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
- 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 102
10 more
- 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
- 2023CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Toolscited by 29
- 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
- 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
- 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 22
- 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
- 2024Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectivescited by 13
- 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
- 2024ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosurescited by 12
- 2025Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generationcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.