mlsanityData to 14 Oct 2026

PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

arXiv 2608.26180 ↗First posted 20 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next-step guidance into a coherent recommendation experience. This changes the unit of evaluation: a fluent response can still fail by ignoring shopper context, contradicting itself across components, or leaving defects too vague to localize. Existing personalization, grounding, and LLM-as-a-judge benchmarks cover pieces of this problem, but they do not define a joint evaluation target for structured shopping-assistant responses. We formulate this missing evaluation target as PACE: Personalized, Actionable, Compositional, and Evidence-grounded evaluation. We instantiate PACE with two artifacts: PACEShop, a benchmark dataset that makes the target measurable through 22,625 controlled records with structured personas, auditable evidence pools, GOOD/BAD labels, and gold defect family and location annotations; and PACEJudge, a training-free judging protocol that makes the target reportable through a structured output contract. Our experiments show that generic judges can recognize broad quality but fail to recover the diagnostic fields required for PACE; PACEShop makes these failures verifiable, and PACEJudge improves persona-source, cross-component, grounding, and family/location closure without retraining, showing that realistic shopping-assistant evaluation requires a task-matched output contract rather than only a stronger backbone or scalar prompt.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

14 of 18 references.

References · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  5. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  6. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  7. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
  8. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  9. 2024Can LLM be a Personalized Judge?cited by 100
  10. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 65
  11. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 48
  12. 2025ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?cited by 27
2 more
  1. 2025Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivitycited by 17
  2. 2025Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judgecited by 4

Built on by

Not cited yet.