mlsanityData to 14 Oct 2026

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation

arXiv 2303.03608 ↗First posted 7 Mar 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. In this work, we develop strong-performing automatic metrics for reference-based summarization evaluation, based on a two-stage evaluation pipeline that first extracts basic information units from one text sequence and then checks the extracted units in another sequence. The metrics we developed include two-stage metrics that can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. We make the developed tools publicly available at https://github.com/Yale-LILY/AutoACU.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

27 of 39 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
15 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  5. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  6. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  9. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  11. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  12. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  13. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  14. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  15. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 8; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  2. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  3. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  4. 2024M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Modelscited by 27
  5. 2024SiReRAG: Indexing Similar and Related Information for Multihop Reasoningcited by 23
  6. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  7. 2023Fair Abstractive Summarization of Diverse Perspectivescited by 17
  8. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4cited by 16
  9. 2025GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checkingcited by 12
  10. 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processingcited by 12
  11. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  12. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
12 more
  1. 2025Evaluating the Evaluators: Are readability metrics good measures of readability?cited by 6
  2. 2023QTSumm: Query-Focused Summarization over Tabular Datacited by 4
  3. 2024Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmarkcited by 4
  4. 2024NovAScore: A New Automated Metric for Evaluating Document Level Noveltycited by 4
  5. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  6. 2025Can Large Language Models be Effective Online Opinion Miners?cited by 3
  7. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  8. 2026CARE: A Conformal Safety Layer for Medical Summarizationcited by 2
  9. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learningcited by 2
  10. 2023The Extractive-Abstractive Axis: Measuring Content "Borrowing" in Generative Language Modelscited by 1
  11. 2025Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarizationcited by 1
  12. 2025Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP Systemcited by 1