# FFCI: A Framework for Interpretable Automatic Evaluation of Summarization

arXiv: https://arxiv.org/abs/2011.13662 · HTML: https://mlsanity.com/paper/2011.13662/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 25 of 25)

- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation](https://mlsanity.com/paper/2108.13134/) (2021)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Learning to Revise References for Faithful Summarization](https://mlsanity.com/paper/2204.10290/) (2022)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation](https://mlsanity.com/paper/2305.01498/) (2023)
- [Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries](https://mlsanity.com/paper/2109.09195/) (2021)
- [APPLS: Evaluating Evaluation Metrics for Plain Language Summarization](https://mlsanity.com/paper/2305.14341/) (2023)
- [Evaluating the Efficacy of Summarization Evaluation across Languages](https://mlsanity.com/paper/2106.01478/) (2021)
- [Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation](https://mlsanity.com/paper/2404.03278/) (2024)
- [Using Similarity to Evaluate Factual Consistency in Summaries](https://mlsanity.com/paper/2409.15090/) (2024)
- [Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods](https://mlsanity.com/paper/2208.13265/) (2022)
- [How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation](https://mlsanity.com/paper/2209.06517/) (2022)
- [Graphmax for Text Generation](https://mlsanity.com/paper/2101.00153/) (2021)
- [PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2203.01769/) (2022)
- [A Critical Look at Meta-evaluating Summarisation Evaluation Metrics](https://mlsanity.com/paper/2409.19507/) (2024)
- [Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English](https://mlsanity.com/paper/2503.17039/) (2025)
- [Just ClozE! A Novel Framework for Evaluating the Factual Consistency Faster in Abstractive Summarization](https://mlsanity.com/paper/2210.02804/) (2022)
- [Evaluating the Efficacy of Length-Controllable Machine Translation](https://mlsanity.com/paper/2305.02300/) (2023)
- [Evaluation of Faithfulness Using the Longest Supported Subsequence](https://mlsanity.com/paper/2308.12157/) (2023)
- [Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages](https://mlsanity.com/paper/2510.07061/) (2025)
- [Leveraging Hierarchical Organization for Medical Multi-document Summarization](https://mlsanity.com/paper/2510.23104/) (2025)
- [Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges](https://mlsanity.com/paper/2605.23970/) (2026)
- [Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation](https://mlsanity.com/paper/2609.35308/) (2026)

## Abstract (arXiv)

In this paper, we propose FFCI, a framework for fine-grained summarization evaluation that comprises four elements: faithfulness (degree of factual consistency with the source), focus (precision of summary content relative to the reference), coverage (recall of summary content relative to the reference), and inter-sentential coherence (document fluency between adjacent sentences). We construct a novel dataset for focus, coverage, and inter-sentential coherence, and develop automatic methods for evaluating each of the four dimensions of FFCI based on cross-comparison of evaluation metrics and model-based evaluation methods, including question answering (QA) approaches, semantic textual similarity (STS), next-sentence prediction (NSP), and scores derived from 19 pre-trained language models. We then apply the developed metrics in evaluating a broad range of summarization models across two datasets, with some surprising findings.