mlsanityData to 14 Oct 2026

Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics

arXiv 2406.15264 ↗First posted 21 Jun 2024cs.IR cs.CLcited by 17

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations." To mitigate this, retrieval-augmented LLMs incorporate citations, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies use faithfulness metrics to estimate citation support automatically but are limited to binary classification, overlooking fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results show no single metric consistently excels across all evaluations, revealing the complexity of assessing fine-grained support. Based on the findings, we provide practical recommendations for developing more effective metrics.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 45 of its 65 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

45 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
32 more
  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  3. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  4. 2019ELI5: Long Form Question Answeringcited by 575
  5. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  6. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  7. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  8. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  9. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  10. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  11. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  12. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  13. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  14. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  15. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  16. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  17. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  18. 2020DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanationcited by 43
  19. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 42
  20. 2023Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectificationcited by 38
  21. 2021GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarizationcited by 26
  22. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  23. 2020Query-controllable Video Summarizationcited by 24
  24. 2023Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attributioncited by 24
  25. 2024Improving Attributed Text Generation of Large Language Models via Preference Learningcited by 23
  26. 2017A Novel Framework for Robustness Analysis of Visual QA Modelscited by 20
  27. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19
  28. 2022Shortcomings of Question Answering Based Factuality Frameworks for Error Localizationcited by 15
  29. 2024QFMTS: Generating Query-Focused Summaries over Multi-Table Inputscited by 9
  30. 2023Causalainer: Causal Explainer for Automatic Video Summarizationcited by 8
  31. 2022BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metricscited by 7
  32. 2023Causal Video Summarizer for Video Explorationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  2. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 34
  3. 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
  4. 2024Correctness is not Faithfulness in RAG Attributionscited by 18
  5. 2024A Novel Evaluation Framework for Image2Text Generationcited by 11
  6. 2024Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Modelscited by 10
  7. 2024Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Modelscited by 5
  8. 2025Large Language Model Sourcing: A Surveycited by 5
  9. 2024Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization using Large Language Modelscited by 4
  10. 2025Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citationcited by 4
  11. 2024A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluationcited by 2
  12. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
5 more
  1. 2024Automated Retinal Image Analysis and Medical Report Generation through Deep Learningcited by 1
  2. 2024Personalized Video Summarization using Text-Based Queries and Conditional Modelingcited by 1
  3. 2025LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learningcited by 1
  4. 2025DeepEyeNet: Generating Medical Report for Retinal Imagescited by 1
  5. 2026A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
  2. 2025CiteCheck: Towards Accurate Citation Faithfulness Detection
  3. 2024On the Capacity of Citation Generation by Large Language Models
  4. 2026How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation
  5. 2026Aligning Large Language Model Behavior with Human Citation Preferences
  6. 2026Universal Pathologies, Conditional Consequences: A Triple-Robustness Analysis of RAG for Multi-Hop Traceability
  7. 2026A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability
  8. 2026DS@GT ARC at LongEval: Citation Integrity and Factual Grounding in Scientific QA