# The Utility of LLMs in Recommender Systems Explanation Evaluation

arXiv: https://arxiv.org/abs/2609.01627 · HTML: https://mlsanity.com/paper/2609.01627/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [Contrasting Linguistic Patterns in Human and LLM-Generated News Text](https://mlsanity.com/paper/2308.09067/) (2023)
- [Counterfactual Explainable Recommendation](https://mlsanity.com/paper/2108.10539/) (2021)
- [CAFE: Coarse-to-Fine Neural Symbolic Reasoning for Explainable Recommendation](https://mlsanity.com/paper/2010.15620/) (2020)
- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [Large Language Models as Evaluators for Recommendation Explanations](https://mlsanity.com/paper/2406.03248/) (2024)
- [Unlocking the Potential of Large Language Models for Explainable Recommendations](https://mlsanity.com/paper/2312.15661/) (2023)
- [On Explaining Recommendations with Large Language Models: A Review](https://mlsanity.com/paper/2411.19576/) (2024)
- [RecExplainer: Aligning Large Language Models for Explaining Recommendation Models](https://mlsanity.com/paper/2311.10947/) (2023)
- [Reinforcement Recommendation Reasoning through Knowledge Graphs for Explanation Path Quality](https://mlsanity.com/paper/2209.04954/) (2022)
- [Post Processing Recommender Systems with Knowledge Graphs for Recency, Popularity, and Diversity of Explanations](https://mlsanity.com/paper/2204.11241/) (2022)
- [Decision Making of Maximizers and Satisficers Based on Collaborative Explanations](https://mlsanity.com/paper/1805.11537/) (2018)
- [Impact of Tone-Aware Explanations in Recommender Systems](https://mlsanity.com/paper/2405.05061/) (2024)
- [MindReader: Recommendation over Knowledge Graph Entities with Explicit User Ratings](https://mlsanity.com/paper/2106.04209/) (2021)
- [The influence of user personality and rating scale features on rating behaviour: an empirical study](https://mlsanity.com/paper/2306.09504/) (2023)
- [Can Offline Metrics Measure Explanation Goals? A Comparative Survey Analysis of Offline Explanation Metrics in Recommender Systems](https://mlsanity.com/paper/2310.14379/) (2023)
- [Review of Explainable Graph-Based Recommender Systems](https://mlsanity.com/paper/2408.00166/) (2024)
- [Whom do Explanations Serve? A Systematic Literature Survey of User Characteristics in Explainable Recommender Systems Evaluation](https://mlsanity.com/paper/2412.14193/) (2024)

## Abstract (arXiv)

Explanations play a crucial role in creating trustworthy recommender systems (RS), yet choosing a good explanation method presents challenges. Many explanation methods exist, but little guidance exists on which is best for which setting. Existing explanation generation methods often produce abstract outputs that require further formatting to become user-friendly, with a seemingly endless pool of options. Running user-based evaluations of all possible options is usually unfeasible, while automated evaluation metrics often either assess only the explainer's abstract output or require comparison with a ground truth, which is generally unavailable. Recent studies have shown that large language models (LLMs) can serve as ``judges'' for explanation evaluation, but their reliability has not yet been thoroughly explored. This paper studies the utility of LLMs in selecting an effective explanation method for a given application. We first explore their ability to generate explanation prototypes given varying information about the RS and the user. Specifically, we generate 18 distinct explanation prototypes, which are subsequently evaluated by 14 LLMs of varying sizes across two temperature settings. We compare these against human ratings derived from a user study. Our results show that while LLMs exhibit human-like rating patterns and achieve moderate rank correlation with human raters, their absolute rating agreement is low and varies substantially by model size and evaluation construct. We derive four practical recommendations: keep explanation-generation prompts concise, prefer larger models for evaluation, pre-test evaluation constructs, and audit explanations for factual accuracy, as neither humans nor LLMs reliably detect non-factual content.