mlsanityData to 14 Oct 2026

SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction

arXiv 2503.01478 ↗First posted 3 Mar 2025cs.CL cs.AI cs.LGcited by 24

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated improved generation performance by incorporating externally retrieved knowledge, a process known as retrieval-augmented generation (RAG). Despite the potential of this approach, existing studies evaluate RAG effectiveness by 1) assessing retrieval and generation components jointly, which obscures retrieval's distinct contribution, or 2) examining retrievers using traditional metrics such as NDCG, which creates a gap in understanding retrieval's true utility in the overall generation process. To address the above limitations, in this work, we introduce an automatic evaluation method that measures retrieval quality through the lens of information gain within the RAG framework. Specifically, we propose Semantic Perplexity (SePer), a metric that captures the LLM's internal belief about the correctness of the retrieved information. We quantify the utility of retrieval by the extent to which it reduces semantic perplexity post-retrieval. Extensive experiments demonstrate that SePer not only aligns closely with human preferences but also offers a more precise and efficient evaluation of retrieval utility across diverse RAG scenarios.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 66 of its 84 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

66 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 65

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,691
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  13. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  14. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  15. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  16. 2019How Can We Know What Language Models Know?cited by 959
  17. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  18. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  19. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  20. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  21. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  22. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  23. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  24. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  25. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  26. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  27. 2023Active Retrieval Augmented Generationcited by 516
  28. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  29. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  30. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  31. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  32. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  33. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  34. 2019Measurement and Fairnesscited by 308
  35. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  36. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  37. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  38. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 15.

24 papers.

Sort:

By area:

  1. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 42
  2. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  4. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 19
  5. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
  6. 2025ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Researchcited by 7
  7. 2026How Many Tools Should an LLM Agent See? A Chance-Corrected Answercited by 7
  8. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  9. 2026ElasticMem: Latent Memory as a Learnable Resource for LLM Agentscited by 4
  10. 2025Redefining Retrieval Evaluation in the Era of LLMscited by 3
  11. 2026Adaptive Debiasing Tsallis Entropy for Test-Time Adaptationcited by 3
  12. 2026Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoningcited by 3
12 more
  1. 2025Simple and Efficient Heterogeneous Temporal Graph Neural Networkcited by 2
  2. 2026Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generationcited by 2
  3. 2026Multimodal Classification via Total Correlation Maximizationcited by 2
  4. 2026The Geometry of Flow-Matching Uncertainty: A Cost-free Uncertainty Proxy and Its Application in Flow-based VLA Failure Detectioncited by 2
  5. 2025Beyond Perplexity: Let the Reader Select Retrieval Summaries via Spectrum Projection Scorecited by 1
  6. 2026Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answeringcited by 1
  7. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  8. 2026Return or Revise? Learning When Revision Helps Retrieval-Augmented QAcited by 1
  9. 2026Beyond Relevance: Utility-Centric Retrieval in the LLM Eracited by 0
  10. 2026Aligning Dense Retrievers with LLM Utility via Distillationcited by 0
  11. 2026Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Mediacited by 0
  12. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Are Large Language Models Good at Utility Judgments?
  2. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  3. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
  4. 2025The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generation
  5. 2025LLM-Specific Utility for Retrieval-Augmented Generation
  6. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmark
  7. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  8. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systems