mlsanityData to 7 Oct 2026

VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

arXiv 2609.00921 ↗First posted 1 Sep 2026cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce VIBE-Bench, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position VIBE-Bench as a focused testbed for advancing preference reasoning beyond semantic matching.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 60 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

29 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Aligning LLMs with Individual Preferences via Interaction4 Oct

Other references · 28

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2020Concept Bottleneck Modelscited by 1,022
  5. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  6. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  7. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  8. 2021Towards Emotional Support Dialog Systemscited by 294
  9. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  10. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  11. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  12. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
16 more
  1. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  2. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  3. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  4. 2024Towards Lifelong Dialogue Agents via Timeline-based Memory Managementcited by 50
  5. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 49
  6. 2023LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordinationcited by 46
  7. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  8. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 44
  9. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversationcited by 31
  10. 2025LaMP-QA: A Benchmark for Personalized Long-form Question Answeringcited by 30
  11. 2025Toward Multi-Session Personalized Conversation: A Large-Scale Dataset and Hierarchical Tree Framework for Implicit Reasoningcited by 21
  12. 2025Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysiscited by 16
  13. 2024Beyond Retrieval: Generating Narratives in Conversational Recommender Systemscited by 8
  14. 2025Enhancing Interpretable Image Classification Through LLM Agents and Conditional Concept Bottleneck Modelscited by 6
  15. 2025Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Modelscited by 4
  16. 2025WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classificationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
  2. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  3. 2026Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  5. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
  6. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
  7. 2026Group Preference Collapse in Personalized Multimodal Large Language Models
  8. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization