# Large Language Models as Evaluators for Recommendation Explanations

arXiv: https://arxiv.org/abs/2406.03248 · HTML: https://mlsanity.com/paper/2406.03248/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Explainable Recommendation: A Survey and New Perspectives](https://mlsanity.com/paper/1804.11192/) (2018)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent](https://mlsanity.com/paper/2007.07422/) (2020)
- [Generate Natural Language Explanations for Recommendation](https://mlsanity.com/paper/2101.03392/) (2021)
- [Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation](https://mlsanity.com/paper/2202.06466/) (2022)
- [Neighborhood-Enhanced Supervised Contrastive Learning for Collaborative Filtering](https://mlsanity.com/paper/2402.11523/) (2024)
- [Collaborative-Enhanced Prediction of Spending on Newly Downloaded Mobile Games under Consumption Uncertainty](https://mlsanity.com/paper/2404.08301/) (2024)

## Built on by (most cited first, 19 of 19)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [OpenEP: Open-Ended Future Event Prediction](https://mlsanity.com/paper/2408.06578/) (2024)
- [Improving LLM-powered Recommendations with Personalized Information](https://mlsanity.com/paper/2502.13845/) (2025)
- [What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text](https://mlsanity.com/paper/2503.04804/) (2025)
- [Automated Review Generation Method Based on Large Language Models](https://mlsanity.com/paper/2407.20906/) (2024)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [An Empirical Evaluation of Large Language Models on Consumer Health Questions](https://mlsanity.com/paper/2501.00208/) (2024)
- [Evaluating the Evaluators: Are readability metrics good measures of readability?](https://mlsanity.com/paper/2508.19221/) (2025)
- [Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendation](https://mlsanity.com/paper/2410.13248/) (2024)
- [GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification](https://mlsanity.com/paper/2603.29112/) (2026)
- [SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA](https://mlsanity.com/paper/2504.07385/) (2025)
- [AI and My Values: User Perceptions of LLMs' Ability to Extract, Embody, and Explain Human Values from Casual Conversations](https://mlsanity.com/paper/2601.22440/) (2026)
- [RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models](https://mlsanity.com/paper/2412.11068/) (2024)
- [CORONA: A Coarse-to-Fine Framework for Graph-based Recommendation with Large Language Models](https://mlsanity.com/paper/2506.17281/) (2025)
- [SAGE: Scalable AI Governance & Evaluation](https://mlsanity.com/paper/2602.07840/) (2026)
- [AgentGR: Semantic-aware Agentic Group Decision-Making Simulator for Group Recommendation](https://mlsanity.com/paper/2605.10367/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [The Utility of LLMs in Recommender Systems Explanation Evaluation](https://mlsanity.com/paper/2609.01627/) (2026)
- [XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?](https://mlsanity.com/paper/2609.09428/) (2026)

## Abstract (arXiv)

The explainability of recommender systems has attracted significant attention in academia and industry. Many efforts have been made for explainable recommendations, yet evaluating the quality of the explanations remains a challenging and unresolved issue. In recent years, leveraging LLMs as evaluators presents a promising avenue in Natural Language Processing tasks (e.g., sentiment classification, information extraction), as they perform strong capabilities in instruction following and common-sense reasoning. However, evaluating recommendation explanatory texts is different from these NLG tasks, as its criteria are related to human perceptions and are usually subjective. In this paper, we investigate whether LLMs can serve as evaluators of recommendation explanations. To answer the question, we utilize real user feedback on explanations given from previous work and additionally collect third-party annotations and LLM evaluations. We design and apply a 3-level meta evaluation strategy to measure the correlation between evaluator labels and the ground truth provided by users. Our experiments reveal that LLMs, such as GPT4, can provide comparable evaluations with appropriate prompts and settings. We also provide further insights into combining human labels with the LLM evaluation process and utilizing ensembles of multiple heterogeneous LLM evaluators to enhance the accuracy and stability of evaluations. Our study verifies that utilizing LLMs as evaluators can be an accurate, reproducible and cost-effective solution for evaluating recommendation explanation texts. Our code is available at https://github.com/Xiaoyu-SZ/LLMasEvaluator.