mlsanityData to 14 Oct 2026

textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluation

arXiv 2402.19457 ↗First posted 29 Feb 2024cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Assessing the quality of summarizers poses significant challenges. In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. We introduce textttCOSMIC as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like textttBERTScore and textttROUGE highlight the competitive performance of textttCOSMIC.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

31 of 50 references.

References · 31

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2015Teaching Machines to Read and Comprehendcited by 1,049
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
19 more
  1. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  3. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  4. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  5. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  6. 2020A Theory of Usable Information Under Computational Constraintscited by 162
  7. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  8. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  9. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  10. 2020Pre-trained Summarization Distillationcited by 71
  11. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 68
  12. 2017Text Summarization Techniques: A Brief Surveycited by 59
  13. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  14. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  15. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  16. 2021Unsupervised Extractive Summarization using Pointwise Mutual Informationcited by 23
  17. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 13
  18. 2022On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluationcited by 7
  19. 2023Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  2. 2024GraphNarrator: Generating Textual Explanations for Graph Neural Networkscited by 7
  3. 2025A Rate-Distortion Framework for Summarizationcited by 3
  4. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  5. 2026Discrete Bridges for Mutual Information Estimationcited by 1
  6. 2025Comparing Approaches to Automatic Summarization in Less-Resourced Languagescited by 0