mlsanityData to 7 Oct 2026

Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation

arXiv 2408.15171 ↗First posted 27 Aug 2024cs.CLcited by 1

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

The use of large language models (LLMs) has significantly increased since the introduction of ChatGPT in 2022, demonstrating their value across various applications. However, a major challenge for enterprise and commercial adoption of LLMs is their tendency to generate inaccurate information, a phenomenon known as "hallucination." This project proposes a method for estimating the factuality of a summary generated by LLMs when compared to a source text. Our approach utilizes Naive Bayes classification to assess the accuracy of the content produced.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 20 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

19 of 20 references.

References · 19

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  5. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  6. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  8. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  9. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  10. 2023Prediction-Powered Inferencecited by 214
  11. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  12. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
7 more
  1. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  2. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  3. 2022Multi-hop Question Answeringcited by 64
  4. 2021BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Textcited by 22
  5. 2023AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generationcited by 10
  6. 2024Generative Information Retrieval Evaluationcited by 7
  7. 2024R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Knowledge-Graph Based RAG System Evaluation Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models
  2. 2024Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output
  3. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
  4. 2023Evaluating Factual Consistency of Summaries with Large Language Models
  5. 2024Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations
  6. 2025Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation
  7. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
  8. 2024Long-form factuality in large language models