mlsanityData to 7 Oct 2026

Question Answering as an Automatic Evaluation Metric for News Article Summarization

arXiv 1906.00318 ↗First posted 2 Jun 2019cs.CL cs.IRcited by 63

Not on the map. Why?

Abstract (arXiv)

Recent work in the field of automatic summarization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrinsic, evaluation metric for this task, Answering Performance for Evaluation of Summaries. APES utilizes recent progress in the field of reading-comprehension to quantify the ability of a summary to answer a set of manually created questions regarding central entities in the source article. We first analyze the strength of this metric by comparing it to known manual evaluation metrics. We then present an end-to-end neural abstractive model that maximizes APES, while increasing ROUGE scores to competitive results.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 17; 2023: 6; 2024: 4; 2025: 10; 2026: 2.

63 papers.

Sort:

By area:

  1. 2021Efficient Attentions for Long Document Summarizationcited by 342
  2. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  3. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  4. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  5. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  6. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  7. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  8. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  9. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  10. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  11. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  12. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
38 more
  1. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  2. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  3. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  4. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  5. 2021Factual Consistency Evaluation for Text Summarization via Counterfactual Estimationcited by 37
  6. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  7. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  8. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  9. 2023MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarizationcited by 30
  10. 2022Learning to Revise References for Faithful Summarizationcited by 28
  11. 2021The Summary Loop: Learning to Write Abstractive Summaries Without Examplescited by 27
  12. 2021Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric Viewcited by 24
  13. 2021RetGen: A Joint framework for Retrieval and Grounded Text Generation Modelingcited by 22
  14. 2023Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesiscited by 20
  15. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
  16. 2024Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checkingcited by 18
  17. 2021Controllable Summarization with Constrained Markov Decision Processcited by 17
  18. 2022Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Modelscited by 17
  19. 2021Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summariescited by 15
  20. 2022Shortcomings of Question Answering Based Factuality Frameworks for Error Localizationcited by 15
  21. 2025REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformationscited by 15
  22. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  23. 2022QASem Parsing: Text-to-text Modeling of QA-based Semanticscited by 12
  24. 2021CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarizationcited by 11
  25. 2025SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactionscited by 11
  26. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  27. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  28. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAGcited by 9
  29. 2021Language Model Augmented Relevance Scorecited by 7
  30. 2021Estimation of Summary-to-Text Inconsistency by Mismatched Embeddingscited by 5
  31. 2021The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generationcited by 5
  32. 2021Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selectioncited by 5
  33. 2023Question-Answering Approach to Evaluating Legal Summariescited by 5
  34. 2023A comprehensive review of automatic text summarization techniques: method, data, evaluation and codingcited by 4
  35. 2022MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplificationcited by 3
  36. 2022Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generationcited by 3
  37. 2025SteLLA: A Structured Grading System Using LLMs with RAGcited by 3
  38. 2022WIDAR -- Weighted Input Document Augmented ROUGEcited by 2

50 most cited of 63 All 63 (JSON)