mlsanityData to 7 Oct 2026

ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

arXiv 2405.18638 ↗First posted 28 May 2024cs.CL cs.AIcited by 15

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the experimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. Furthermore, the evaluation should differentiate the capabilities and weaknesses of increasingly powerful large language models -- which requires effective test sets. The scalability of human evaluation is also crucial to wider adoption. Hence, to design an effective human evaluation system in the age of generative NLP, we propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 4 of its 113 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

4 of 113 references.

References · 4

  1. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  3. 2023Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and Considerations in Real-World Scenarioscited by 4
  4. 2023Principles from Clinical Research for NLP Model Generalizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  2. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
  3. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  4. 2025LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Datacited by 10
  5. 2026EarthVL: A Progressive Earth Vision-Language Understanding and Generation Frameworkcited by 5
  6. 2025TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notescited by 3
  7. 2025Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Promptscited by 3
  8. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  9. 2025Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Modelscited by 2
  10. 2025A Novel, Human-in-the-Loop Computational Grounded Theory Framework for Big Social Datacited by 2
  11. 2025The Impact of Foundational Models on Patient-Centric e-Health Systemscited by 0
  12. 2026AgriLens: Semantic Retrieval in Agricultural Texts Using Topic Modeling and Language Modelscited by 0
3 more
  1. 2026How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluationcited by 0
  2. 2026Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wildcited by 0
  3. 2026Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  2. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gap
  3. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  4. 2025Aligning Black-box Language Models with Human Judgments
  5. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  6. 2023Human Feedback is not Gold Standard
  7. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  8. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function