mlsanityData to 7 Oct 2026

Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

arXiv 2606.23915 ↗First posted 22 Jun 2026cs.CL cs.IR cs.LGcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Practice often treats automatic metrics for attribution in LLM retrieval-augmented generation as interchangeable. We audit eight automatic scorers -- lexical, embedding, and BERTScore baselines alongside entailment/grounding-trained models (clean and FEVER NLI, the checker MiniCheck) -- across three evaluation constructs (provenance/topicality, generated-answer attribution, and fact-check entailment), asking whether any scorer transfers: stays within the 95% confidence interval of the best audited scorer on every dataset of a multi-dataset construct. In the construct with the most multi-dataset human-labeled coverage -- generated-answer attribution (AttributionBench's four source datasets, n = 1,610, with independent HAGRID, n = 2,150) -- none of the audited automatic scorers does: the per-dataset metric rankings invert (Kendall tau = -0.64, p = 0.031 on AttributedQA vs. LFQA), and an off-the-shelf NLI scorer that is best on short-claim AttributedQA (AUROC 0.90) collapses to AUROC 0.53 (chance) on long-form LFQA, where BERTScore wins (0.91); the reversal persists under the tested truncation settings. This instability has a concrete decision cost: a naive "best-on-average" rule for choosing an evaluator fails leave-one-dataset-out (mean held-out regret 0.172 AUROC, worse than fixing one scorer), so metric choice should be validated on the target dataset rather than assumed from performance elsewhere. A prompt-based LLM judge avoids the chance-level collapses the automatic scorers suffer (no LFQA collapse) but is not uniformly best, ~100x costlier, and non-deterministic -- relocating, not removing, the validation burden.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 39 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

27 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents16 Apr

Other references · 26

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  11. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  12. 2022Teaching language models to support answers with verified quotescited by 290
14 more
  1. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  2. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  3. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  4. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  5. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  6. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  7. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  8. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  9. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 40
  10. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  11. 2025Lessons from Training Grounded LLMs with Verifiable Rewardscited by 8
  12. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  13. 2026GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidencecited by 6
  14. 2026Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Attribute or Abstain: Large Language Models as Long Document Assistants
  2. 2024Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models
  3. 2026How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation
  4. 2025SAFE: Improving LLM Systems using Sentence-Level In-generation Attribution
  5. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
  6. 2024Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation
  7. 2025Source Attribution in Retrieval-Augmented Generation
  8. 2024LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation