mlsanityData to 7 Oct 2026

The Personalization Paradox: Semantic Loss vs. Reasoning Gains in Agentic AI Q&A

arXiv 2512.04343 ↗First posted 4 Dec 2025cs.IRcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

AIVisor, an agentic retrieval-augmented LLM for student advising, was used to examine how personalization affects system performance across multiple evaluation dimensions. Using twelve authentic advising questions intentionally designed to stress lexical precision, we compared ten personalized and non-personalized system configurations and analyzed outcomes with a Linear Mixed-Effects Model across lexical (BLEU, ROUGE-L), semantic (METEOR, BERTScore), and grounding (RAGAS) metrics. Results showed a consistent trade-off: personalization reliably improved reasoning quality and grounding, yet introduced a significant negative interaction on semantic similarity, driven not by poorer answers but by the limits of current metrics, which penalize meaningful personalized deviations from generic reference texts. This reveals a structural flaw in prevailing LLM evaluation methods, which are ill-suited for assessing user-specific responses. The fully integrated personalized configuration produced the highest overall gains, suggesting that personalization can enhance system effectiveness when evaluated with appropriate multidimensional metrics. Overall, the study demonstrates that personalization produces metric-dependent shifts rather than uniform improvements and provides a methodological foundation for more transparent and robust personalization in agentic AI.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 33 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

9 of 33 references.

References · 9

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  3. 2023A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspectivecited by 33
  4. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 30
  5. 2023Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Functioncited by 7
  6. 2024Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needscited by 4
  7. 2024Applying Fine-Tuned LLMs for Reducing Data Needs in Load Profile Analysiscited by 2
  8. 2025A Survey of Affective Recommender Systems: Modeling Attitudes, Emotions, and Moods for Personalizationcited by 2
  9. 2023"Do it my way!": Impact of Customizations on Trust perceptions in Human-Robot Collaborationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
  2. 2025Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrieval
  3. 2026FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
  4. 2025AdvisingWise: Supporting Academic Advising in Higher Education Settings Through a Human-in-the-Loop Multi-Agent Framework
  5. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
  6. 2025A Survey of Personalization: From RAG to Agent
  7. 2025Are Generative AI Agents Effective Personalized Financial Advisors?
  8. 2026APeB: Benchmarking Personalization Ability of Large Language Model Agents