mlsanityData to 7 Oct 2026

FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge

arXiv 2305.08281 ↗First posted 14 May 2023cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Evaluating the factual consistency of automatically generated summaries is essential for the progress and adoption of reliable summarization systems. Despite recent advances, existing factuality evaluation models are not robust, being especially prone to entity and relation errors in new domains. We propose FactKB, a simple new approach to factuality evaluation that is generalizable across domains, in particular with respect to entities and relations. FactKB is based on language models pretrained using facts extracted from external knowledge bases. We introduce three types of complementary factuality pretraining objectives based on direct entity facts, facts grounded in auxiliary knowledge about entities, and facts constructed compositionally through knowledge base walks. The resulting factuality evaluation model achieves state-of-the-art performance on two in-domain news summarization benchmarks as well as on three out-of-domain scientific literature datasets. Further analysis of FactKB shows improved ability to detect erroneous entities and relations in summaries and is robust and generalizable across domains.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 71 of its 114 references that are in the map. First posted on arXiv 14 May 2023.

Builds on

71 of 114 references.

References · 71

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
38 more
  1. 2019Text Summarization with Pretrained Encoderscited by 553
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  4. 2019Composition-based Multi-Relational Graph Convolutional Networkscited by 395
  5. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  6. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  7. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  8. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  9. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  10. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  12. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  13. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  14. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  15. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  16. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  17. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  18. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  19. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  20. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  21. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  22. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  23. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  24. 2020Tensor Decompositions for temporal knowledge base completioncited by 115
  25. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  26. 2021COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemiccited by 100
  27. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  28. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  29. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  30. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  31. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  32. 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 54
  33. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  34. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  35. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  36. 2022FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representationscited by 41
  37. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  38. 2022Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queriescited by 39

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 27; 2025: 9; 2026: 5.

55 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  3. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  4. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  5. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  6. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  7. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  8. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  9. 2024Self-Taught Evaluatorscited by 70
  10. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 49
  11. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  12. 2023QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checkingcited by 26
38 more
  1. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 26
  2. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  3. 2024DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinationscited by 23
  4. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graphcited by 22
  5. 2024Graph-Based Retriever Captures the Long Tail of Biomedical Knowledgecited by 20
  6. 2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
  7. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  8. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  9. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  10. 2024The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Modelscited by 17
  11. 2024DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Modelscited by 15
  12. 2024FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoningcited by 12
  13. 2023Sentiment Analysis through LLM Negotiationscited by 11
  14. 2023AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generationcited by 10
  15. 2023Context-aware Decoding Reduces Hallucination in Query-focused Summarizationcited by 10
  16. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  17. 2025Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Modelscited by 9
  18. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  19. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  20. 2024FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Modelscited by 7
  21. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  22. 2025HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Modelscited by 7
  23. 2024Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Modelscited by 6
  24. 2023Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generationcited by 5
  25. 2024Hallucination Diversity-Aware Active Learning for Text Summarizationcited by 5
  26. 2024Attention with Dependency Parsing Augmentation for Fine-Grained Attributioncited by 5
  27. 2025Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluationcited by 5
  28. 2025ZINA: Multimodal Fine-grained Hallucination Detection and Editingcited by 5
  29. 2023P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Modelscited by 4
  30. 2024KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entitiescited by 4
  31. 2025DiFaR: Enhancing Multimodal Misinformation Detection with Diverse, Factual, and Relevant Rationalescited by 2
  32. 2026Detecting Contextual Hallucinations in LLMs with Frequency-Aware Attentioncited by 2
  33. 2024Open Artificial Knowledgecited by 1
  34. 2024Pre-trained Language Models Return Distinguishable Probability Distributions to Unfaithfully Hallucinated Textscited by 1
  35. 2024Measuring the Groundedness of Legal Question-Answering Systemscited by 1
  36. 2026Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decodingcited by 1
  37. 2026Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalizationcited by 1
  38. 2024A Knowledge-Injected Curriculum Pretraining Framework for Question Answeringcited by 0

50 most cited of 55 All 55 (JSON)