# Beyond Utility: Evaluating LLM as Recommender

arXiv: https://arxiv.org/abs/2411.00331 · HTML: https://mlsanity.com/paper/2411.00331/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2024.

## Built on by (most cited first, 15 of 15)

- [Collab-REC: An LLM-based Agentic Framework for Balancing Recommendations in Tourism](https://mlsanity.com/paper/2508.15030/) (2025)
- [EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection](https://mlsanity.com/paper/2505.17654/) (2025)
- [Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation](https://mlsanity.com/paper/2503.05493/) (2025)
- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [HADSF: Aspect Aware Semantic Control for Explainable Recommendation](https://mlsanity.com/paper/2510.26994/) (2025)
- [Tree of Preferences for Diversified Recommendation](https://mlsanity.com/paper/2601.02386/) (2025)
- [Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation](https://mlsanity.com/paper/2602.08873/) (2026)
- [Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Data](https://mlsanity.com/paper/2511.04541/) (2025)
- [Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems](https://mlsanity.com/paper/2602.02582/) (2026)
- [LLM-HYPER: Generative CTR Modeling for Cold-Start Ad Personalization via LLM-Based Hypernetworks](https://mlsanity.com/paper/2604.12096/) (2026)
- [Personalized Recommendation Tool Learning via Autonomous Language Agents](https://mlsanity.com/paper/2607.19739/) (2026)
- [Selective LLM-Guided Regularization for Enhancing Recommendation Models](https://mlsanity.com/paper/2512.21526/) (2025)
- [SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2604.09553/) (2026)
- [AgentGR: Semantic-aware Agentic Group Decision-Making Simulator for Group Recommendation](https://mlsanity.com/paper/2605.10367/) (2026)
- [Attacking and Defending Multi-Agent Collaborative Filtering Systems Through Connectivity](https://mlsanity.com/paper/2608.03272/) (2026)

## Abstract (arXiv)

With the rapid development of Large Language Models (LLMs), recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate nonexistent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.