# Hierarchical Evaluation Framework: Best Practices for Human Evaluation

arXiv: https://arxiv.org/abs/2310.01917 · HTML: https://mlsanity.com/paper/2310.01917/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Trading Off Diversity and Quality in Natural Language Generation](https://mlsanity.com/paper/2004.10450/) (2020)
- [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2304.01816/) (2023)
- [NAREOR: The Narrative Reordering Problem](https://mlsanity.com/paper/2104.06669/) (2021)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [Translating Hanja Historical Documents to Contemporary Korean and English](https://mlsanity.com/paper/2205.10019/) (2022)
- [A Data-centric Framework for Improving Domain-specific Machine Reading Comprehension Datasets](https://mlsanity.com/paper/2304.00483/) (2023)
- [Building Extractive Question Answering System to Support Human-AI Health Coaching Model for Sleep Domain](https://mlsanity.com/paper/2305.19707/) (2023)

## Built on by (most cited first, 5 of 5)

- [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems](https://mlsanity.com/paper/2605.02122/) (2026)
- [Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2410.12380/) (2024)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)

## Abstract (arXiv)

Human evaluation plays a crucial role in Natural Language Processing (NLP) as it assesses the quality and relevance of developed systems, thereby facilitating their enhancement. However, the absence of widely accepted human evaluation metrics in NLP hampers fair comparisons among different systems and the establishment of universal assessment standards. Through an extensive analysis of existing literature on human evaluation metrics, we identified several gaps in NLP evaluation methodologies. These gaps served as motivation for developing our own hierarchical evaluation framework. The proposed framework offers notable advantages, particularly in providing a more comprehensive representation of the NLP system's performance. We applied this framework to evaluate the developed Machine Reading Comprehension system, which was utilized within a human-AI symbiosis model. The results highlighted the associations between the quality of inputs and outputs, underscoring the necessity to evaluate both components rather than solely focusing on outputs. In future work, we will investigate the potential time-saving benefits of our proposed framework for evaluators assessing NLP systems.