# FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations

arXiv: https://arxiv.org/abs/2204.06508 · HTML: https://mlsanity.com/paper/2204.06508/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 68 references that are in the map.
- First posted on arXiv 13 Apr 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Leveraging Graph to Improve Abstractive Multi-Document Summarization](https://mlsanity.com/paper/2005.10043/) (2020)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://mlsanity.com/paper/1806.09835/) (2018)
- [A Graph-to-Sequence Model for AMR-to-Text Generation](https://mlsanity.com/paper/1805.02473/) (2018)
- [Improving Factual Consistency of Abstractive Summarization via Question Answering](https://mlsanity.com/paper/2105.04623/) (2021)
- [Leveraging Abstract Meaning Representation for Knowledge Base Question Answering](https://mlsanity.com/paper/2012.01707/) (2020)
- [Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization](https://mlsanity.com/paper/2108.13684/) (2021)
- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [AMR Parsing via Graph-Sequence Iterative Inference](https://mlsanity.com/paper/2004.05572/) (2020)
- [Toward Abstractive Summarization Using Semantic Representations](https://mlsanity.com/paper/1805.10399/) (2018)

## Built on by (most cited first, 30 of 41)

- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge](https://mlsanity.com/paper/2305.08281/) (2023)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Faithfulness-Aware Decoding Strategies for Abstractive Summarization](https://mlsanity.com/paper/2303.03278/) (2023)
- [GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework](https://mlsanity.com/paper/2407.10793/) (2024)
- [Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective](https://mlsanity.com/paper/2411.14258/) (2024)
- [FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction](https://mlsanity.com/paper/2403.02270/) (2024)
- [Zero-shot Faithful Factual Error Correction](https://mlsanity.com/paper/2305.07982/) (2023)
- [How Far are We from Robust Long Abstractive Summarization?](https://mlsanity.com/paper/2210.16732/) (2022)
- [Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks](https://mlsanity.com/paper/2402.17630/) (2024)
- [The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models](https://mlsanity.com/paper/2404.05904/) (2024)
- [Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation](https://mlsanity.com/paper/2405.03085/) (2024)
- [DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2403.00896/) (2024)
- [A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization](https://mlsanity.com/paper/2303.03948/) (2023)
- [Localizing Factual Inconsistencies in Attributable Text Generation](https://mlsanity.com/paper/2410.07473/) (2024)
- [GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking](https://mlsanity.com/paper/2502.16514/) (2025)
- [AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation](https://mlsanity.com/paper/2311.09521/) (2023)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [A review of faithfulness metrics for hallucination assessment in Large Language Models](https://mlsanity.com/paper/2501.00269/) (2024)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization](https://mlsanity.com/paper/2305.14548/) (2023)
- [Evaluating LLMs' Assessment of Mixed-Context Hallucination Through the Lens of Summarization](https://mlsanity.com/paper/2503.01670/) (2025)
- [Linking Surface Facts to Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2310.14909/) (2023)
- [Hallucination Diversity-Aware Active Learning for Text Summarization](https://mlsanity.com/paper/2404.01588/) (2024)
- [Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection](https://mlsanity.com/paper/2305.14163/) (2023)
- [P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Models](https://mlsanity.com/paper/2311.09741/) (2023)
- [PatentEval: Understanding Errors in Patent Generation](https://mlsanity.com/paper/2406.06589/) (2024)
- [Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning](https://mlsanity.com/paper/2507.19586/) (2025)
- [Summarization with Graphical Elements](https://mlsanity.com/paper/2204.07551/) (2022)
- [MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations](https://mlsanity.com/paper/2505.14101/) (2025)

## Abstract (arXiv)

Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FactGraph encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph connectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%. Furthermore, FactGraph improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.