mlsanityData to 14 Oct 2026

Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation

arXiv 1904.05929 ↗First posted 11 Apr 2019cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Conducting a manual evaluation is considered an essential part of summary evaluation methodology. Traditionally, the Pyramid protocol, which exhaustively compares system summaries to references, has been perceived as very reliable, providing objective scores. Yet, due to the high cost of the Pyramid method and the required expertise, researchers resorted to cheaper and less thorough manual evaluation methods, such as Responsiveness and pairwise comparison, attainable via crowdsourcing. We revisit the Pyramid approach, proposing a lightweight sampling-based version that is crowdsourcable. We analyze the performance of our method in comparison to original expert-based Pyramid evaluations, showing higher correlation relative to the common Responsiveness method. We release our crowdsourced Summary-Content-Units, along with all crowdsourcing scripts, for future evaluations.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 7; 2024: 9; 2025: 3; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  4. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  5. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  6. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 77
  7. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  8. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  9. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  10. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 40
  11. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 38
  12. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
27 more
  1. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  2. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  3. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  4. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  5. 2024FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Documentcited by 15
  6. 2022Revisiting text decomposition methods for NLI-based factuality scoring of summariescited by 14
  7. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  8. 2022Controlled Text Reductioncited by 10
  9. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  10. 2025Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generationcited by 8
  11. 2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
  12. 2024WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citationscited by 7
  13. 2024The Power of Summary-Source Alignmentscited by 7
  14. 2021BEAMetrics: A Benchmark for Language Generation Evaluation Evaluationcited by 6
  15. 2023Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Taskscited by 6
  16. 2024On the Role of Summary Content Units in Text Summarization Evaluationcited by 5
  17. 2024Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challengescited by 5
  18. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  19. 2022DATScore: Evaluating Translation with Data Augmented Translationscited by 2
  20. 2026When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generationcited by 2
  21. 2023Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarizationcited by 1
  22. 2025OpenFActScore: Open-Source Atomic Evaluation of Factuality in Text Generationcited by 1
  23. 2021Towards Human-Free Automatic Quality Evaluation of German Summarizationcited by 0
  24. 2021GeSERA: General-domain Summary Evaluation by Relevance Analysiscited by 0
  25. 2022Novel Chapter Abstractive Summarization using Spinal Tree Aware Sub-Sentential Content Selectioncited by 0
  26. 2026TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarizationcited by 0
  27. 2026SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarizationcited by 0