mlsanityData to 7 Oct 2026

SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes

arXiv 2212.09305 ↗First posted 19 Dec 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Is it possible to train a general metric for evaluating text generation quality without human annotated ratings? Existing learned metrics either perform unsatisfactorily across text generation tasks or require human ratings for training on specific tasks. In this paper, we propose SESCORE2, a self-supervised approach for training a model-based metric for text generation evaluation. The key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus. The primary advantage of the SESCORE2 is its ease of extension to many other languages while providing reliable severity estimation. We evaluate SESCORE2 and previous methods on four text generation tasks across three languages. SESCORE2 outperforms unsupervised metric PRISM on four text generation evaluation benchmarks, with a Kendall improvement of 0.078. Surprisingly, SESCORE2 even outperforms the supervised BLEURT and COMET on multiple text generation tasks. The code and data are available at https://github.com/xu1998hz/SEScore2.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

26 of 51 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
14 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2020Evaluation of Text Generation: A Surveycited by 275
  3. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  4. 2015A Neural Conversational Modelcited by 195
  5. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  8. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  9. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  10. 2021BERTTune: Fine-Tuning Neural Machine Translation with BERTScorecited by 18
  11. 2022Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesiscited by 18
  12. 2017JESC: Japanese-English Subtitle Corpuscited by 16
  13. 2022Contextual Representation Learning beyond Masked Language Modelingcited by 16
  14. 2018Extracting Parallel Paragraphs from Common Crawlcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 2.

9 papers.

  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  2. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  3. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  4. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  5. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  6. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  7. 2023N-Critics: Self-Refinement of Large Language Models with Ensemble of Criticscited by 8
  8. 2025Searching the Internet for Challenging Benchmarks at Scalecited by 2
  9. 2024Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systemscited by 1