# T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics

arXiv: https://arxiv.org/abs/2212.05726 · HTML: https://mlsanity.com/paper/2212.05726/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [TransQuest at WMT2020: Sentence-Level Direct Assessment](https://mlsanity.com/paper/2010.05318/) (2020)
- [Evaluating the Efficacy of Summarization Evaluation across Languages](https://mlsanity.com/paper/2106.01478/) (2021)
- [RoBLEURT Submission for the WMT2021 Metrics Task](https://mlsanity.com/paper/2204.13352/) (2022)
- [ParaCotta: Synthetic Multilingual Paraphrase Corpora from the Most Diverse Translation Sample Pair](https://mlsanity.com/paper/2205.04651/) (2022)

## Built on by (most cited first, 19 of 19)

- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://mlsanity.com/paper/2303.15621/) (2023)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [FormalAlign: Automated Alignment Evaluation for Autoformalization](https://mlsanity.com/paper/2410.10135/) (2024)
- [Better Instruction-Following Through Minimum Bayes Risk](https://mlsanity.com/paper/2410.02902/) (2024)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation](https://mlsanity.com/paper/2309.12455/) (2023)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)
- [DeltaScore: Fine-Grained Story Evaluation with Perturbations](https://mlsanity.com/paper/2303.08991/) (2023)
- [Pretraining Language Models Using Translationese](https://mlsanity.com/paper/2403.13638/) (2024)
- [Exploring Automatic Evaluation Methods based on a Decoder-based LLM for Text Generation](https://mlsanity.com/paper/2310.11026/) (2023)
- [ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluation](https://mlsanity.com/paper/2504.02106/) (2025)
- [Rate-Resolved Text Evaluation with Masked Diffusion Language Models](https://mlsanity.com/paper/2610.15667/) (2026)

## Abstract (arXiv)

Modern embedding-based metrics for evaluation of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs are of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the probabilities of a generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text. In this paper, we present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as the backbone. We perform an extensive empirical comparison with other existing metrics on 5 datasets, 19 languages and 280 systems, demonstrating the utility of our method. Experimental results show that: T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level. We release our code and models at https://github.com/qinyiwei/T5Score.