mlsanityData to 7 Oct 2026

Towards a Unified Multi-Dimensional Evaluator for Text Generation

arXiv 2210.07197 ↗First posted 13 Oct 2022cs.CLcited by 264

Not on the map. Why?

Abstract (arXiv)

Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository (https://github.com/maszhongming/UniEval).

Key facts. Cited by 264 papers in the map (data through 7 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

38 of 55 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2018Neural Network Acceptability Judgmentscited by 952
26 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  6. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  7. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  8. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  9. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  12. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  13. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  14. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  15. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  16. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  17. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  18. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  19. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  20. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  21. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  22. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  23. 2019Searching for Effective Neural Extractive Summarization: What Works and What's Nextcited by 38
  24. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  25. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  26. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 66; 2024: 100; 2025: 63; 2026: 29.

264 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2023Active Retrieval Augmented Generationcited by 506
  5. 2023GPTScore: Evaluate as You Desirecited by 357
  6. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  7. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  8. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  11. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  12. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
38 more
  1. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  2. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  3. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  4. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  5. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  8. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  9. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  10. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  11. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  12. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  13. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  14. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  15. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  16. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 54
  17. 2023Calibrating LLM-Based Evaluatorcited by 53
  18. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  19. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  20. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  21. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  22. 2023Fusing Models with Complementary Expertisecited by 45
  23. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
  24. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  25. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  26. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  27. 2023Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvementscited by 42
  28. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  29. 2023Chainpoll: A high efficacy method for LLM hallucination detectioncited by 39
  30. 2023Instruct and Extract: Instruction Tuning for On-Demand Information Extractioncited by 35
  31. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  32. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  33. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  34. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  35. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literaturecited by 29
  36. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  37. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  38. 2023Summarizing Multiple Documents with Conversational Structure for Meta-Review Generationcited by 27

50 most cited of 264 All 264 (JSON)