mlsanityData to 14 Oct 2026

Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation

arXiv 2504.05220 ↗First posted 7 Apr 2025cs.IR cs.AI cs.CLcited by 5

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems, aiming to reduce dependence on costly human annotations. We address the gap between retrieval relevance and generative utility by employing LLMs to annotate document utility. To effectively utilize multiple positive samples per query, we introduce a novel loss that maximizes their summed marginal likelihood. Using the Qwen-2.5-32B model, we annotate utility on the MS MARCO dataset and conduct retrieval experiments on MS MARCO and BEIR, as well as RAG experiments on MS MARCO QA, NQ, and HotpotQA. Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics. Furthermore, combining LLM annotations with just 20% of human labels achieves performance comparable to using full human annotations. Our study offers a comprehensive approach to utilizing LLM annotations for initializing QA systems on new corpora.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 40 of its 64 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

40 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization5 May

Other references · 39

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
27 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  4. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  5. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  6. 2020Overview of the TREC 2019 deep learning trackcited by 286
  7. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  8. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  9. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  10. 2021Overview of the TREC 2020 deep learning trackcited by 196
  11. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  12. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  13. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  14. 2023Large language models can accurately predict searcher preferencescited by 131
  15. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  16. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  17. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  18. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  19. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  20. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  21. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  22. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  23. 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
  24. 2024Are Large Language Models Good at Utility Judgments?cited by 22
  25. 2024LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?cited by 11
  26. 2023RRAML: Reinforced Retrieval Augmented Machine Learningcited by 6
  27. 2024An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Injecting External Knowledge into the Reasoning Process Enhances Retrieval-Augmented Generationcited by 3
  2. 2026Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generationcited by 2
  3. 2026Training Dense Retrievers with Multiple Positive Passagescited by 1
  4. 2026AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generationcited by 1
  5. 2026UsefulBench: Towards Decision-Useful Information as a Target for Information Retrievalcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-Specific Utility for Retrieval-Augmented Generation
  2. 2025Redefining Retrieval Evaluation in the Era of LLMs
  3. 2026Beyond Relevance: Utility-Centric Retrieval in the LLM Era
  4. 2025Less LLM, More Documents: Searching for Improved RAG
  5. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  6. 2025Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
  7. 2024DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
  8. 2025Insight-RAG: Enhancing LLMs with Insight-Driven Augmentation