# SumQE: a BERT-based Summary Quality Estimation Model

arXiv: https://arxiv.org/abs/1909.00578 · HTML: https://mlsanity.com/paper/1909.00578/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2019.

## Built on by (most cited first, 17 of 17)

- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [On the Limitations of Reference-Free Evaluations of Generated Text](https://mlsanity.com/paper/2210.12563/) (2022)
- [Finding a Balanced Degree of Automation for Summary Evaluation](https://mlsanity.com/paper/2109.11503/) (2021)
- [A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy](https://mlsanity.com/paper/2106.13945/) (2021)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation](https://mlsanity.com/paper/2103.10918/) (2021)
- [Decision-Focused Summarization](https://mlsanity.com/paper/2109.06896/) (2021)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation](https://mlsanity.com/paper/2101.11298/) (2021)
- [Diagnostic Captioning: A Survey](https://mlsanity.com/paper/2101.07299/) (2021)
- [Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings](https://mlsanity.com/paper/2104.05156/) (2021)
- [How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation](https://mlsanity.com/paper/2209.06517/) (2022)
- [A comprehensive review of automatic text summarization techniques: method, data, evaluation and coding](https://mlsanity.com/paper/2301.03403/) (2023)
- [Exploring the Correlation between Human and Machine Evaluation of Simultaneous Speech Translation](https://mlsanity.com/paper/2406.10091/) (2024)
- [Consistency and Coherence from Points of Contextual Similarity](https://mlsanity.com/paper/2112.11638/) (2021)
- [Towards Human-Free Automatic Quality Evaluation of German Summarization](https://mlsanity.com/paper/2105.06027/) (2021)
- [Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics](https://mlsanity.com/paper/2210.14174/) (2022)

## Abstract (arXiv)

We propose SumQE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, without involving comparison with human references. SumQE achieves very high correlations with human ratings, outperforming simpler models addressing these linguistic aspects. Predictions of the SumQE model can be used for system development, and to inform users of the quality of automatically produced summaries and other types of generated text.