mlsanityData to 7 Oct 2026

ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders

arXiv 2602.16938 ↗First posted 18 Feb 2026cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

The promise of LLM-based user simulators to improve conversational AI is hindered by a critical "realism gap," leading to systems that are optimized for simulated interactions, but may fail to perform well in the real world. We introduce ConvApparel, a new dataset of human-AI conversations designed to address this gap. Its unique dual-agent data collection protocol -- using both "good" and "bad" recommenders -- enables counterfactual validation by capturing a wide spectrum of user experiences, enriched with first-person annotations of user satisfaction. We propose a comprehensive validation framework that combines statistical alignment, a human-likeness score, and counterfactual validation to test for generalization. Our experiments reveal a significant realism gap across all simulators. However, the framework also shows that data-driven simulators outperform a prompted baseline, particularly in counterfactual validation where they adapt more realistically to unseen behaviors, suggesting they embody more robust, if imperfect, user models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 80 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

35 of 80 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  9. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  10. 2018Towards Deep Conversational Recommendationscited by 174
  11. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  12. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
23 more
  1. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  2. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  3. 2020INSPIRED: Toward Sociable Recommendation Dialog Systemscited by 91
  4. 2025Large Language Models Pass the Turing Testcited by 48
  5. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 46
  6. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  7. 2024LLM-Powered User Simulator for Recommender Systemcited by 38
  8. 2023Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue Systemcited by 32
  9. 2024USimAgent: Large Language Models for Simulating Search Userscited by 31
  10. 2023User Simulation for Evaluating Information Access Systemscited by 30
  11. 2023PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulatorcited by 28
  12. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
  13. 2023An In-depth Investigation of User Response Simulation for Conversational Searchcited by 20
  14. 2023In-Context Learning User Simulators for Task-Oriented Dialog Systemscited by 19
  15. 2025User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluationcited by 16
  16. 2024Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Datasetcited by 15
  17. 2024LLM Roleplay: Simulating Human-Chatbot Interactioncited by 15
  18. 2025PAARS: Persona Aligned Agentic Retail Shopperscited by 14
  19. 2023UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systemscited by 11
  20. 2024SUBER: An RL Environment with Simulated Human Behavior for Recommender Systemscited by 11
  21. 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Makingcited by 4
  22. 2024Towards a Formal Characterization of User Simulation Objectives in Conversational Information Accesscited by 3
  23. 2025Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branchingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  2. 2026τ-Rec: A Verifiable Benchmark for Agentic Recommender Systemscited by 2
  3. 2026Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomescited by 1
  4. 2026Controllable User Simulationcited by 0
  5. 2026Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agentscited by 0
  6. 2026Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluationcited by 0
  7. 2026Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentationcited by 0
  8. 2026"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulatorscited by 0
  9. 2026Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillationcited by 0
  10. 2026On Evaluating and Improving Conversational Agents in Productioncited by 0