# ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

arXiv: https://arxiv.org/abs/2405.18638 · HTML: https://mlsanity.com/paper/2405.18638/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 4 of its 113 references that are in the map.
- First posted on arXiv 28 May 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 4 of 4)

- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and Considerations in Real-World Scenarios](https://mlsanity.com/paper/2306.14373/) (2023)
- [Principles from Clinical Research for NLP Model Generalization](https://mlsanity.com/paper/2311.03663/) (2023)

## Built on by (most cited first, 15 of 15)

- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge](https://mlsanity.com/paper/2410.03775/) (2024)
- [A Scalable Framework for Evaluating Health Language Models](https://mlsanity.com/paper/2503.23339/) (2025)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework](https://mlsanity.com/paper/2601.02783/) (2026)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes](https://mlsanity.com/paper/2503.20648/) (2025)
- [A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellness](https://mlsanity.com/paper/2512.08936/) (2025)
- [Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models](https://mlsanity.com/paper/2503.16581/) (2025)
- [A Novel, Human-in-the-Loop Computational Grounded Theory Framework for Big Social Data](https://mlsanity.com/paper/2506.06083/) (2025)
- [The Impact of Foundational Models on Patient-Centric e-Health Systems](https://mlsanity.com/paper/2507.21882/) (2025)
- [AgriLens: Semantic Retrieval in Agricultural Texts Using Topic Modeling and Language Models](https://mlsanity.com/paper/2601.08283/) (2026)
- [How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation](https://mlsanity.com/paper/2601.09084/) (2026)
- [Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild](https://mlsanity.com/paper/2604.16304/) (2026)
- [Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering](https://mlsanity.com/paper/2606.07523/) (2026)

## Abstract (arXiv)

In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the experimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. Furthermore, the evaluation should differentiate the capabilities and weaknesses of increasingly powerful large language models -- which requires effective test sets. The scalability of human evaluation is also crucial to wider adoption. Hence, to design an effective human evaluation system in the age of generative NLP, we propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.