mlsanityData to 14 Oct 2026

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation

arXiv 2305.13194 ↗First posted 22 May 2023cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems and 4 datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2022). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

44 of 67 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
32 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  2. 2015Teaching Machines to Read and Comprehendcited by 1,049
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  9. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  10. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  11. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  12. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  13. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  14. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2017Challenges in Data-to-Document Generationcited by 226
  17. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  18. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  19. 2023A Categorical Archive of ChatGPT Failurescited by 168
  20. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  21. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  22. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  23. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  24. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  25. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  26. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  27. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  28. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  29. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  30. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  31. 2021DialFact: A Benchmark for Fact-Checking in Dialoguecited by 62
  32. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 23; 2025: 11; 2026: 4.

42 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  2. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  3. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  4. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  5. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  6. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  7. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  8. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 31
  9. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  10. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 26
  11. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  12. 2025Cost-Optimal Active AI Model Evaluationcited by 19
30 more
  1. 2024Comparing Hallucination Detection Metrics for Multilingual Generationcited by 18
  2. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
  3. 2024METAL: Towards Multilingual Meta-Evaluationcited by 15
  4. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  5. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  6. 2023Low-Rank Adaptation for Multilingual Summarization: An Empirical Studycited by 13
  7. 2024A Modular Approach for Multimodal Summarization of TV Showscited by 13
  8. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  9. 2024Bayesian Prediction-Powered Inferencecited by 8
  10. 2024Hierarchical Indexing for Retrieval-Augmented Opinion Summarizationcited by 7
  11. 2023Fusion-Eval: Integrating Assistant Evaluators with LLMscited by 6
  12. 2024GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviewscited by 6
  13. 2025How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucinationcited by 5
  14. 2025Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluatorscited by 5
  15. 2025MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMscited by 5
  16. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 5
  17. 2024LCFO: Long Context and Long Form Output Dataset and Benchmarkingcited by 4
  18. 2024Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Modelscited by 4
  19. 2025LLM-as-a-qualitative-judge: automating error analysis in natural language generationcited by 3
  20. 2023Automatic Personalized Impression Generation for PET Reports Using Large Language Modelscited by 2
  21. 2024German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Datasetcited by 2
  22. 2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
  23. 2025Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond Englishcited by 2
  24. 2025Reward Models are Metrics in a Trench Coatcited by 2
  25. 2025Fine-Grained Detection of Context-Grounded Hallucinations Using LLMscited by 1
  26. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1
  27. 2024Zero-shot Factual Consistency Evaluation Across Domainscited by 0
  28. 2025Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languagescited by 0
  29. 2026SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metricscited by 0
  30. 2026Does task decomposition improve automatic NLG evaluation?cited by 0