mlsanityData to 14 Oct 2026

AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs

arXiv 2410.19692 ↗First posted 25 Oct 2024cs.CL cs.AI cs.IRcited by 7

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Generating diverse and effective clarifying questions is crucial for improving query understanding and retrieval performance in open-domain conversational search (CS) systems. We propose AGENT-CQ (Automatic GENeration, and evaluaTion of Clarifying Questions), an end-to-end LLM-based framework addressing the challenges of scalability and adaptability faced by existing methods that rely on manual curation or template-based approaches. AGENT-CQ consists of two stages: a generation stage employing LLM prompting strategies to generate clarifying questions, and an evaluation stage (CrowdLLM) that simulates human crowdsourcing judgments using multiple LLM instances to assess generated questions and answers based on comprehensive quality metrics. Extensive experiments on the ClariQ dataset demonstrate CrowdLLM's effectiveness in evaluating question and answer quality. Human evaluation and CrowdLLM show that the AGENT-CQ - generation stage, consistently outperforms baselines in various aspects of question and answer quality. In retrieval-based evaluation, LLM-generated questions significantly enhance retrieval effectiveness for both BM25 and cross-encoder models compared to human-generated questions.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 62 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

36 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024BASES: Large-scale Web Search User Simulation with Large Language Model based Agents27 Feb

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  8. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  11. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 139
  12. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
23 more
  1. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 116
  2. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
  3. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  4. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  6. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  7. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 79
  8. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  9. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  10. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  11. 2024Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learningcited by 51
  12. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 48
  13. 2023LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluationcited by 34
  14. 2024PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrievalcited by 30
  15. 2024Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Searchcited by 24
  16. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulationcited by 24
  17. 2020Analyzing and Learning from User Interactions for Search Clarificationcited by 23
  18. 2021Analysing Mixed Initiatives and Search Strategies during Conversational Searchcited by 13
  19. 2021User Engagement Prediction for Clarification in Searchcited by 12
  20. 2021The Great Misalignment Problem in Human Evaluation of NLP Methodscited by 8
  21. 2024Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Searchcited by 8
  22. 2022MIMICS-Duo: Offline & Online Evaluation of Search Clarificationcited by 7
  23. 2024Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfactioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  2. 2025Characterizing Web Search in The Age of Generative AIcited by 5
  3. 2025Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understandingcited by 3
  4. 2025Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrievalcited by 1
  5. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Searchcited by 0
  6. 2026VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policiescited by 0
  7. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
  2. 2024Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations
  3. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  4. 2025Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems
  5. 2025A Survey of Large Language Model Agents for Question Answering
  6. 2025RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering
  7. 2023Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate