mlsanityData to 14 Oct 2026

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

arXiv 1909.02622 ↗First posted 5 Sep 2019cs.CLcited by 418

Not on the map. Why?

Abstract (arXiv)

A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.

Key facts. Cited by 418 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 57; 2022: 101; 2023: 87; 2024: 84; 2025: 47; 2026: 42.

418 papers.

Sort:

By area:

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  8. 2023GPTScore: Evaluate as You Desirecited by 361
  9. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  10. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  11. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  12. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
38 more
  1. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  2. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  3. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  4. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  5. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  6. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  7. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  8. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  9. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  10. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  11. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  12. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  13. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 84
  14. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  15. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 77
  16. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  17. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  18. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  19. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  20. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  21. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  22. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  23. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  24. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  25. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  26. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  27. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 57
  28. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  29. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  30. 2023Calibrating LLM-Based Evaluatorcited by 53
  31. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  32. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  33. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  34. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  35. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  36. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  37. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  38. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46

50 most cited of 418 All 418 (JSON)