# ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions

arXiv: https://arxiv.org/abs/2407.00942 · HTML: https://mlsanity.com/paper/2407.00942/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 93 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia](https://mlsanity.com/paper/2305.14292/) (2023)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [On the (In)Effectiveness of Large Language Models for Chinese Text Correction](https://mlsanity.com/paper/2307.09007/) (2023)
- [Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.09738/) (2024)
- [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024)
- [LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles](https://mlsanity.com/paper/2308.10855/) (2023)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrieval](https://mlsanity.com/paper/2203.03367/) (2022)
- [QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correction](https://mlsanity.com/paper/2403.11886/) (2024)
- [Contrastive Learning with Hard Negative Entities for Entity Set Expansion](https://mlsanity.com/paper/2204.07789/) (2022)
- [Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark](https://mlsanity.com/paper/2402.19248/) (2024)
- [In-Context Learning User Simulators for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2306.00774/) (2023)
- [SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding](https://mlsanity.com/paper/2308.10529/) (2023)

## Built on by (most cited first, 10 of 10)

- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation](https://mlsanity.com/paper/2605.20204/) (2026)
- [EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration](https://mlsanity.com/paper/2601.16489/) (2026)
- [Towards Personalized Conversational Sales Agents: Contextual User Profiling for Strategic Action](https://mlsanity.com/paper/2504.08754/) (2025)
- [IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents](https://mlsanity.com/paper/2601.06676/) (2026)
- [PSCon: Product Search Through Conversations](https://mlsanity.com/paper/2502.13881/) (2025)
- [Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning](https://mlsanity.com/paper/2609.06027/) (2026)
- [FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writing](https://mlsanity.com/paper/2511.22883/) (2025)
- [From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities](https://mlsanity.com/paper/2608.26950/) (2026)
- [From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scale](https://mlsanity.com/paper/2609.05758/) (2026)

## Abstract (arXiv)

This paper introduces the task of product demand clarification within an e-commercial scenario, where the user commences the conversation with ambiguous queries and the task-oriented agent is designed to achieve more accurate and tailored product searching by asking clarification questions. To address this task, we propose ProductAgent, a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. Specifically, we develop the agent with strategies for product feature summarization, query generation, and product retrieval. Furthermore, we propose the benchmark called PROCLARE to evaluate the agent's performance both automatically and qualitatively with the aid of a LLM-driven user simulator. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns, where user demands become gradually more explicit and detailed. All the source codes will be released after the review anonymity period.