# Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis

arXiv: https://arxiv.org/abs/2506.01262 · HTML: https://mlsanity.com/paper/2506.01262/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [Teach LLMs to Personalize -- An Approach inspired by Writing Education](https://mlsanity.com/paper/2308.07968/) (2023)
- [Improving Dialog Systems for Negotiation with Personality Modeling](https://mlsanity.com/paper/2010.09954/) (2020)
- [Pchatbot: A Large-Scale Dataset for Personalized Chatbot](https://mlsanity.com/paper/2009.13284/) (2020)
- [Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Persona](https://mlsanity.com/paper/2305.11482/) (2023)
- [Large Language Models for User Interest Journeys](https://mlsanity.com/paper/2305.15498/) (2023)
- [Editing Personality for Large Language Models](https://mlsanity.com/paper/2310.02168/) (2023)
- [Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstration](https://mlsanity.com/paper/2012.15375/) (2020)
- [Evaluating the Performance of Large Language Models via Debates](https://mlsanity.com/paper/2406.11044/) (2024)

## Built on by (most cited first, 17 of 17)

- [ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context](https://mlsanity.com/paper/2603.01357/) (2026)
- [Your Language Model Secretly Contains Personality Subnetworks](https://mlsanity.com/paper/2602.07164/) (2026)
- [AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios](https://mlsanity.com/paper/2508.19988/) (2025)
- [Responsible Agentic AI Requires Explicit Provenance](https://mlsanity.com/paper/2605.17169/) (2026)
- [Contextualized Visual Personalization in Vision-Language Models](https://mlsanity.com/paper/2602.03454/) (2026)
- [HingeMem: Boundary Guided Long-Term Memory with Query Adaptive Retrieval for Scalable Dialogues](https://mlsanity.com/paper/2604.06845/) (2026)
- [LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs](https://mlsanity.com/paper/2608.05246/) (2026)
- [PREFINE: Personalized Story Generation via Simulated User Critics and User-Specific Rubric Generation](https://mlsanity.com/paper/2510.21721/) (2025)
- [LikeBench: Evaluating Subjective Likability in LLMs for Personalization](https://mlsanity.com/paper/2512.13077/) (2025)
- [What to Remember, What to Reveal: Privacy-Aware Memory for Conversational Agents](https://mlsanity.com/paper/2608.16551/) (2026)
- [Hypotheses-Guided Self Distillation for Continual Personalization](https://mlsanity.com/paper/2609.00251/) (2026)
- [SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information](https://mlsanity.com/paper/2608.10692/) (2026)
- [VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences](https://mlsanity.com/paper/2609.00921/) (2026)
- [PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations](https://mlsanity.com/paper/2609.09664/) (2026)
- [One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction](https://mlsanity.com/paper/2609.21626/) (2026)
- [Unlocking Latent Personalization in LLMs](https://mlsanity.com/paper/2609.33182/) (2026)
- [PersonaFactor: Preference Factors with Dynamic Selection for LLM Personalization](https://mlsanity.com/paper/2610.16372/) (2026)

## Abstract (arXiv)

Personalized AI assistants, a hallmark of the human-like capabilities of Large Language Models (LLMs), are a challenging application that intertwines multiple problems in LLM research. Despite the growing interest in the development of personalized assistants, the lack of an open-source conversational dataset tailored for personalization remains a significant obstacle for researchers in the field. To address this research gap, we introduce HiCUPID, a new benchmark to probe and unleash the potential of LLMs to deliver personalized responses. Alongside a conversational dataset, HiCUPID provides a Llama-3.2-based automated evaluation model whose assessment closely mirrors human preferences. We release our dataset, evaluation model, and code at https://github.com/12kimih/HiCUPID.