# Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge

arXiv: https://arxiv.org/abs/2508.08777 · HTML: https://mlsanity.com/paper/2508.08777/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings](https://mlsanity.com/paper/2503.15620/) (2025)
- [Cold-Start Recommendation towards the Era of Large Language Models (LLMs): A Comprehensive Survey and Roadmap](https://mlsanity.com/paper/2501.01945/) (2025)
- [Large Language Models as Data Augmenters for Cold-Start Item Recommendation](https://mlsanity.com/paper/2402.11724/) (2024)
- [LLMs for User Interest Exploration in Large-scale Recommendation Systems](https://mlsanity.com/paper/2405.16363/) (2024)
- [Current Challenges and Future Directions in Podcast Information Access](https://mlsanity.com/paper/2106.09227/) (2021)
- [User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems](https://mlsanity.com/paper/2504.05522/) (2025)
- [REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives](https://mlsanity.com/paper/2503.11924/) (2025)

## Built on by (most cited first, 6 of 6)

- [Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback](https://mlsanity.com/paper/2602.12612/) (2026)
- [Conf-Profile: A Confidence-Driven Reasoning Paradigm for Label-Free User Profiling](https://mlsanity.com/paper/2509.18864/) (2025)
- [UserGPT Technical Report](https://mlsanity.com/paper/2605.08766/) (2026)
- [A Cascaded Generative Approach for e-Commerce Recommendations](https://mlsanity.com/paper/2605.11118/) (2026)
- [From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation](https://mlsanity.com/paper/2608.11493/) (2026)
- [A Behavioral Trait Leaks into Preferences: Diagnosing Trait Interference in LLM User Simulators](https://mlsanity.com/paper/2609.25572/) (2026)

## Abstract (arXiv)

Evaluating personalized recommendations remains a central challenge, especially in long-form audio domains like podcasts, where traditional offline metrics suffer from exposure bias and online methods such as A/B testing are costly and operationally constrained. In this paper, we propose a novel framework that leverages Large Language Models (LLMs) as offline judges to assess the quality of podcast recommendations in a scalable and interpretable manner. Our two-stage profile-aware approach first constructs natural-language user profiles distilled from 90 days of listening history. These profiles summarize both topical interests and behavioral patterns, serving as compact, interpretable representations of user preferences. Rather than prompting the LLM with raw data, we use these profiles to provide high-level, semantically rich context-enabling the LLM to reason more effectively about alignment between a user's interests and recommended episodes. This reduces input complexity and improves interpretability. The LLM is then prompted to deliver fine-grained pointwise and pairwise judgments based on the profile-episode match. In a controlled study with 47 participants, our profile-aware judge matched human judgments with high fidelity and outperformed or matched a variant using raw listening histories. The framework enables efficient, profile-aware evaluation for iterative testing and model selection in recommender systems.