# ChatShop: Interactive Information Seeking with Language Agents

arXiv: https://arxiv.org/abs/2404.09911 · HTML: https://mlsanity.com/paper/2404.09911/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 23 of 23)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [OpenAgents: An Open Platform for Language Agents in the Wild](https://mlsanity.com/paper/2310.10634/) (2023)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Deep Reinforcement Learning with a Natural Language Action Space](https://mlsanity.com/paper/1511.04636/) (2015)
- [MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming](https://mlsanity.com/paper/2310.06500/) (2023)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning](https://mlsanity.com/paper/1606.02560/) (2016)
- [Decision-Oriented Dialogue for Human-AI Collaboration](https://mlsanity.com/paper/2305.20076/) (2023)
- [Multi-Stage Episodic Control for Strategic Exploration in Text Games](https://mlsanity.com/paper/2201.01251/) (2022)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations](https://mlsanity.com/paper/2004.14530/) (2020)
- [Interactive Machine Comprehension with Information Seeking Agents](https://mlsanity.com/paper/1908.10449/) (2019)
- [Towards Information-Seeking Agents](https://mlsanity.com/paper/1612.02605/) (2016)
- [Know your audience: specializing grounded language models with listener subtraction](https://mlsanity.com/paper/2206.08349/) (2022)

## Built on by (most cited first, 24 of 24)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents](https://mlsanity.com/paper/2512.07850/) (2025)
- [The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Markets](https://mlsanity.com/paper/2506.00073/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants](https://mlsanity.com/paper/2601.18225/) (2026)
- [A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains](https://mlsanity.com/paper/2508.15832/) (2025)
- [VoiceAgentEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Voice-Agent Evaluation of Xbench's Professional-Aligned Series](https://mlsanity.com/paper/2510.21244/) (2025)
- [Modeling Distinct Human Interaction in Web Agents](https://mlsanity.com/paper/2602.17588/) (2026)
- [SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce](https://mlsanity.com/paper/2602.01443/) (2026)
- [InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedback](https://mlsanity.com/paper/2502.15027/) (2025)
- [Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information](https://mlsanity.com/paper/2508.11252/) (2025)
- [SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents](https://mlsanity.com/paper/2605.19219/) (2026)
- [WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation](https://mlsanity.com/paper/2607.06118/) (2026)
- [AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics](https://mlsanity.com/paper/2608.27818/) (2026)
- [Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent](https://mlsanity.com/paper/2609.38604/) (2026)
- [Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agents](https://mlsanity.com/paper/2504.08739/) (2025)
- [Beyond expert users: agents should help users construct preferences, not just elicit them](https://mlsanity.com/paper/2606.30863/) (2026)
- [CLEAT: Learning Value-Guided Clarification for Interactive Agents](https://mlsanity.com/paper/2610.12655/) (2026)

## Abstract (arXiv)

The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent evaluation. We analyze a popular web shopping task designed to test language agents' ability to perform strategic exploration and discover that it can be reformulated and solved as a single-turn retrieval task without the need for interactive information seeking. This finding encourages us to rethink realistic constraints on information access that would necessitate strategic information seeking. We then redesign the task to introduce a notion of task ambiguity and the role of a shopper, serving as a dynamic party with whom the agent strategically interacts in an open-ended conversation to make informed decisions. Our experiments demonstrate that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information through multi-turn interactions. Additionally, we show that large language model-simulated shoppers serve as a good proxy for real human shoppers, revealing similar error patterns in agents.