mlsanityData to 7 Oct 2026

RankME: Reliable Human Ratings for Natural Language Generation

arXiv 1803.05928 ↗First posted 15 Mar 2018cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 12; 2024: 13; 2025: 7; 2026: 8.

67 papers.

Sort:

By area:

  1. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  4. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  5. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  6. 2023Human Feedback is not Gold Standardcited by 82
  7. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  8. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  9. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  10. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  11. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  12. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
38 more
  1. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  2. 2021TellMeWhy: A Dataset for Answering Why-Questions in Narrativescited by 27
  3. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  4. 2022The Authenticity Gap in Human Evaluationcited by 24
  5. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  6. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  7. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  8. 2022Czech Grammar Error Correction with a Large and Diverse Corpuscited by 17
  9. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  10. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 17
  11. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  12. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  13. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  14. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  15. 2021AutoChart: A Dataset for Chart-to-Text Generation Taskcited by 12
  16. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 12
  17. 2024DHP Benchmark: Are LLMs Good NLG Evaluators?cited by 12
  18. 2022Risk-graded Safety for Handling Medical Queries in Conversational AIcited by 11
  19. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 11
  20. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  21. 2026Pearmut: Human Evaluation of Translation Made Trivialcited by 11
  22. 2022Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisonscited by 10
  23. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  24. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  25. 2023Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discoverycited by 10
  26. 2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
  27. 2023Understanding Practices around Computational News Discovery Tools in the Domain of Science Journalismcited by 7
  28. 2022WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphscited by 6
  29. 2023Revisiting Sentence Union Generation as a Testbed for Text Consolidationcited by 6
  30. 2024CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systemscited by 5
  31. 2025Deep Literature Survey Automation with an Iterative Workflowcited by 5
  32. 2021Affective Decoding for Empathetic Response Generationcited by 4
  33. 2022On the Effectiveness of Automated Metrics for Text Generation Systemscited by 4
  34. 2023First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Modelscited by 4
  35. 2021On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputscited by 3
  36. 2024How Much Annotation is Needed to Compare Summarization Models?cited by 3
  37. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  38. 2025From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentarycited by 3

50 most cited of 67 All 67 (JSON)