mlsanityData to 7 Oct 2026

SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

arXiv 2005.03724 ↗First posted 7 May 2020cs.CL cs.IRcited by 69

Not on the map. Why?

Abstract (arXiv)

We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18-39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/acl20-ref-free-eval.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 17; 2023: 18; 2024: 9; 2025: 6; 2026: 1.

69 papers.

Sort:

By area:

  1. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  4. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  6. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  7. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  8. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  9. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  10. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  11. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  12. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
38 more
  1. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  2. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  3. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  4. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
  5. 2021A Survey on Multi-modal Summarizationcited by 26
  6. 2021RefSum: Refactoring Neural Summarizationcited by 25
  7. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  8. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  9. 2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
  10. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  11. 2022Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metricscited by 18
  12. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  13. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  14. 2023Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects -- A Surveycited by 18
  15. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  16. 2023OpinSummEval: Revisiting Automated Evaluation for Opinion Summarizationcited by 13
  17. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 12
  18. 2021Re-evaluating Word Mover's Distancecited by 10
  19. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  20. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  21. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  22. 2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
  23. 2021Estimation of Summary-to-Text Inconsistency by Mismatched Embeddingscited by 5
  24. 2024Large Language Models as Evaluators for Scientific Synthesiscited by 5
  25. 2021Hone as You Read: A Practical Type of Interactive Summarizationcited by 4
  26. 2021RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisationcited by 4
  27. 2023A comprehensive review of automatic text summarization techniques: method, data, evaluation and codingcited by 4
  28. 2023MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Typescited by 4
  29. 2022Entity-driven Fact-aware Abstractive Summarization of Biomedical Literaturecited by 3
  30. 2023UMSE: Unified Multi-scenario Summarization Evaluationcited by 3
  31. 2023Correction of Errors in Preference Ratings from Automated Metrics for Text Generationcited by 3
  32. 2023GNAT: A General Narrative Alignment Toolcited by 3
  33. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  34. 2024Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?cited by 3
  35. 2022WIDAR -- Weighted Input Document Augmented ROUGEcited by 2
  36. 2022PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarizationcited by 2
  37. 2022Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Codecited by 2
  38. 2022DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freelycited by 2

50 most cited of 69 All 69 (JSON)