A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Recent advances in Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users. Yet, fundamental questions about their capabilities, limitations, and paths forward remain open. This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence. To that end, we systematically analyze LLM-driven Conversational Agents by organizing their capabilities into three primary dimensions: (i) Reasoning - logical, systematic thinking inspired by human intelligence for decision making, (ii) Monitor - encompassing self-awareness and user interaction monitoring, and (iii) Control - focusing on tool utilization and policy following. Building upon this, we introduce a novel taxonomy by classifying recent work on Conversational Agents around our proposed desideratum. We identify critical research gaps and outline key directions, including realistic evaluations, long-term multi-turn reasoning skills, self-evolution capabilities, collaborative and multi-agent task completion, personalization, and proactivity. This work aims to provide a structured foundation, highlight existing limitations, and offer insights into potential future research directions for Conversational Agents, ultimately advancing progress toward Artificial General Intelligence (AGI). We maintain a curated repository of papers at: https://github.com/emrecanacikgoz/awesome-conversational-agents.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 125 of its 140 references that are in the map. First posted on arXiv 7 Apr 2025.
Builds on
125 of 140 references.
Closest ancestors · 24
Same direction, newest first
- 2025Agentic Knowledgeable Self-awareness4 Apr
- 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems31 Mar
- 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents3 Mar
- 2025Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model12 Feb
- 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training20 Jan
- 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuning3 Jan
- 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents1 Nov
- 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
- 2024xLAM: A Family of Large Action Models to Empower AI Agent Systems5 Sep
- 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents21 Jun
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
- 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement17 Jun
12 more
- 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation17 May
- 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models19 Mar
- 2024Large Multimodal Agents: A Survey23 Feb
- 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling16 Feb
- 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents14 Feb
- 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
- 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
- 2023FireAct: Toward Language Agent Fine-tuning9 Oct
- 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
- 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
- 2023Cognitive Architectures for Language Agents5 Sep
- 2023A Survey on Large Language Model based Autonomous Agents22 Aug
Other references · 101
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
38 more
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
- 2025s1: Simple test-time scalingcited by 1,311
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
- 2023Reasoning with Language Model is Planning with World Modelcited by 846
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
- 2024Executable Code Actions Elicit Better LLM Agentscited by 537
- 2023LaMP: When Large Language Models Meet Personalizationcited by 425
- 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
- 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
- 2023Tool Learning with Foundation Modelscited by 302
- 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
- 2021Towards Emotional Support Dialog Systemscited by 294
- 2023Large Language Models as Tool Makerscited by 293
- 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
- 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
- 2024TextGrad: Automatic "Differentiation" via Textcited by 250
- 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
- 2023Do Large Language Models Know What They Don't Know?cited by 238
- 2024Tool Learning with Large Language Models: A Surveycited by 221
- 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
- 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
- 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
- 2022TALM: Tool Augmented Language Modelscited by 197
- 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
- 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
- 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
- 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
50 most cited of 101 All 101 (JSON)
Built on by
Citing papers per year: 2025: 4; 2026: 5.
9 papers.
- 2025A Survey of Context Engineering for Large Language Modelscited by 120
- 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 19
- 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisonscited by 10
- 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learningcited by 7
- 2026MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignmentcited by 2
- 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judgingcited by 1
- 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoningcited by 0
- 2026Listening Alone, Understanding Together: Collaborative Context Recovery for Privacy-Aware AIcited by 0
- 2026CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perceptioncited by 0
- 2026CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception9 Aug 2026
- 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging11 Jul 2026
- 2026MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment15 Apr 2026
- 2026Listening Alone, Understanding Together: Collaborative Context Recovery for Privacy-Aware AI14 Apr 2026
- 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning15 Feb 2026
- 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning15 Dec 2025
- 2025Goal Alignment in LLM-Based User Simulators for Conversational AI27 Jul 2025
- 2025A Survey of Context Engineering for Large Language Models17 Jul 2025
- 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons28 Apr 2025
Similar, not cited
Close in wording, no citation either way
- 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
- 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems
- 2025Conversational Alignment with Artificial Intelligence in Context
- 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- 2025Small Language Models are the Future of Agentic AI
- 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
- 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems
- 2025Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap