mlsanityData to 14 Oct 2026

ChatShop: Interactive Information Seeking with Language Agents

arXiv 2404.09911 ↗First posted 15 Apr 2024cs.CLcited by 24

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent evaluation. We analyze a popular web shopping task designed to test language agents' ability to perform strategic exploration and discover that it can be reformulated and solved as a single-turn retrieval task without the need for interactive information seeking. This finding encourages us to rethink realistic constraints on information access that would necessitate strategic information seeking. We then redesign the task to introduce a notion of task ambiguity and the role of a shopper, serving as a dynamic party with whom the agent strategically interacts in an open-ended conversation to make informed decisions. Our experiments demonstrate that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information through multi-turn interactions. Additionally, we show that large language model-simulated shoppers serve as a good proxy for real human shoppers, revealing similar error patterns in agents.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 40 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

23 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  9. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  10. 2023Eliciting Human Preferences with Language Modelscited by 92
  11. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  12. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
10 more
  1. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 71
  2. 2016Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Accesscited by 36
  3. 2016Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learningcited by 29
  4. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  5. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  6. 2021Open-domain clarification question generation without question examplescited by 20
  7. 2020Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversationscited by 19
  8. 2019Interactive Machine Comprehension with Information Seeking Agentscited by 13
  9. 2016Towards Information-Seeking Agentscited by 4
  10. 2022Know your audience: specializing grounded language models with listener subtractioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 9.

24 papers.

Sort:

By area:

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  2. 2025Agentic Large Language Models, a surveycited by 108
  3. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  4. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  5. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 46
  6. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  7. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 33
  8. 2025SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agentscited by 21
  9. 2025The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Marketscited by 15
  10. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  11. 2026ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistantscited by 10
  12. 2025A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domainscited by 7
12 more
  1. 2025VoiceAgentEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Voice-Agent Evaluation of Xbench's Professional-Aligned Seriescited by 5
  2. 2026Modeling Distinct Human Interaction in Web Agentscited by 5
  3. 2026SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commercecited by 4
  4. 2025InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedbackcited by 1
  5. 2025Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Informationcited by 1
  6. 2026SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agentscited by 1
  7. 2026WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluationcited by 1
  8. 2026AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamicscited by 1
  9. 2026Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intentcited by 1
  10. 2025Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agentscited by 0
  11. 2026Beyond expert users: agents should help users construct preferences, not just elicit themcited by 0
  12. 2026CLEAT: Learning Value-Guided Clarification for Interactive Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses
  2. 2026RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce
  3. 2025InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context
  4. 2026Clarify User Expertise: Towards Proactive Conversational Agents Tailoring Responses to User Proficiency
  5. 2026Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
  6. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation
  7. 2026Value of Information: A Framework for Human-Agent Communication
  8. 2026Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning