mlsanityData to 7 Oct 2026

APeB: Benchmarking Personalization Ability of Large Language Model Agents

arXiv 2607.03162 ↗First posted 3 Jul 2026cs.AI cs.HCcited by 0

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 53 of its 92 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

53 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agents17 Nov
  2. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions7 Jul
  3. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics1 May
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 49

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  12. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
37 more
  1. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  2. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 213
  3. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  4. 2025OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignmentcited by 167
  5. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  6. 2018Learning Tree-based Deep Model for Recommender Systemscited by 99
  7. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  8. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  9. 2019A Deep Look into Neural Ranking Models for Information Retrievalcited by 79
  10. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  11. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 58
  12. 2025A Survey on LLM-powered Agents for Recommender Systemscited by 49
  13. 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 47
  14. 2023Pre-train, Prompt and Recommendation: A Comprehensive Survey of Language Modelling Paradigm Adaptations in Recommender Systemscited by 45
  15. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 44
  16. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  17. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  18. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  19. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 26
  20. 2025OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Searchcited by 24
  21. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systemscited by 23
  22. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  23. 2023When Search Meets Recommendation: Learning Disentangled Search Representation for Recommendationcited by 17
  24. 2023KuaiSAR: A Unified Search And Recommendation Datasetcited by 14
  25. 2024A Comprehensive Review of Recommender Systems: Transitioning from Theory to Practicecited by 13
  26. 2024UniSAR: Modeling User Transition Behaviors between Search and Recommendationcited by 12
  27. 2025AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platformscited by 12
  28. 2022Modeling User Behavior with Graph Convolution for Personalized Product Searchcited by 9
  29. 2023JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactionscited by 8
  30. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 8
  31. 2025SessionIntentBench: A Multi-task Inter-session Intention-shift Modeling Benchmark for E-commerce Customer Behavior Understandingcited by 6
  32. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  33. 2022IHGNN: Interactive Hypergraph Neural Network for Personalized Product Searchcited by 5
  34. 2025Unified Representation Learning for Multi-Intent Diversity and Behavioral Uncertainty in Recommender Systemscited by 4
  35. 2025Harnessing Multimodal Large Language Models for Personalized Product Search with Query-aware Refinementcited by 3
  36. 2022Sequential Recommendation with User Evolving Preference Decompositioncited by 2
  37. 2026Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
  2. 2026Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
  3. 2026From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents
  4. 2025A Survey of Personalization: From RAG to Agent
  5. 2026AdaptArena: Evaluating Test-Time Personalization of Web Agents
  6. 2025SPARK: Search Personalization via Agent-Driven Retrieval and Knowledge-sharing
  7. 2026PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails
  8. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants