mlsanityData to 7 Oct 2026

Unifying Human and Statistical Evaluation for Natural Language Generation

arXiv 1904.02792 ↗First posted 4 Apr 2019cs.CL cs.AI stat.MLcited by 127

Not on the map. Why?

Abstract (arXiv)

How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is human- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 28; 2022: 31; 2023: 24; 2024: 15; 2025: 14; 2026: 14.

127 papers.

Sort:

By area:

  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  2. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  3. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  4. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  5. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  6. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  7. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  8. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  9. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  10. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  11. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  12. 2022Truncation Sampling as Language Model Desmoothingcited by 121
38 more
  1. 2022Human heuristics for AI-generated language are flawedcited by 104
  2. 2022Evaluating Human-Language Model Interactioncited by 90
  3. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  4. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  5. 2020Text Generation by Learning from Demonstrationscited by 64
  6. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  7. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  8. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  9. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  10. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  11. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  12. 2021What Context Features Can Transformer Language Models Use?cited by 49
  13. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  14. 2023Evaluating Open-QA Evaluationcited by 35
  15. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  16. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  17. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  18. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  19. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  20. 2021Focus Attention: Promoting Faithfulness and Diversity in Summarizationcited by 30
  21. 2023HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysiscited by 30
  22. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  23. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  24. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  25. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  26. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  27. 2023How Far Can We Extract Diverse Perspectives from Large Language Models?cited by 22
  28. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  29. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  30. 2022Why is constrained neural language generation particularly challenging?cited by 19
  31. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  32. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  33. 2023DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detectioncited by 18
  34. 2023On the Efficacy of Sampling Adapterscited by 17
  35. 2024Evaluating Diversity in Automatic Poetry Generationcited by 17
  36. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  37. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  38. 2023Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSAcited by 15

50 most cited of 127 All 127 (JSON)