# HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems

arXiv: https://arxiv.org/abs/2601.19197 · HTML: https://mlsanity.com/paper/2601.19197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [A Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2207.12515/) (2022)
- [Evaluating ChatGPT as a Recommender System: A Rigorous Approach](https://mlsanity.com/paper/2309.03613/) (2023)
- [On Explaining Recommendations with Large Language Models: A Review](https://mlsanity.com/paper/2411.19576/) (2024)
- [Large Language Model Enhanced Recommender Systems: A Survey](https://mlsanity.com/paper/2412.13432/) (2024)
- [Revealing Potential Biases in LLM-Based Recommender Systems in the Cold Start Setting](https://mlsanity.com/paper/2508.20401/) (2025)

## Abstract (arXiv)

The integration of Large Language Models (LLMs) into recommendation systems has introduced unprecedented capabilities for natural language understanding, explanation generation, and conversational interactions. However, existing evaluation methodologies focus predominantly on traditional accuracy metrics, failing to capture the multifaceted human-centered qualities that determine the real-world user experience. We introduce framework (Human-centered Evaluation for LLM-powered recoMmenders), a comprehensive evaluation framework that systematically assesses LLM-powered recommender systems across five human-centered dimensions: textitIntent Alignment, textitExplanation Quality, textitInteraction Naturalness, textitTrust & Transparency, and textitFairness & Diversity. Through extensive experiments involving three state-of-the-art LLM-based recommenders (GPT-4, LLaMA-3.1, and P5) across three domains (movies, books, and restaurants), and rigorous evaluation by 12 domain experts using 847 recommendation scenarios, we demonstrate that framework reveals critical quality dimensions invisible to traditional metrics. Our results show that while GPT-4 achieves superior explanation quality (4.21/5.0) and interaction naturalness (4.35/5.0), it exhibits a significant popularity bias (Gini coefficient 0.73) compared to traditional collaborative filtering (0.58). We release framework as an open-source toolkit to advance human-centered evaluation practices in the recommender systems community.