mlsanityData to 7 Oct 2026

A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions

arXiv 2504.16939 ↗First posted 7 Apr 2025cs.AI cs.CLcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users. Yet, fundamental questions about their capabilities, limitations, and paths forward remain open. This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence. To that end, we systematically analyze LLM-driven Conversational Agents by organizing their capabilities into three primary dimensions: (i) Reasoning - logical, systematic thinking inspired by human intelligence for decision making, (ii) Monitor - encompassing self-awareness and user interaction monitoring, and (iii) Control - focusing on tool utilization and policy following. Building upon this, we introduce a novel taxonomy by classifying recent work on Conversational Agents around our proposed desideratum. We identify critical research gaps and outline key directions, including realistic evaluations, long-term multi-turn reasoning skills, self-evolution capabilities, collaborative and multi-agent task completion, personalization, and proactivity. This work aims to provide a structured foundation, highlight existing limitations, and offer insights into potential future research directions for Conversational Agents, ultimately advancing progress toward Artificial General Intelligence (AGI). We maintain a curated repository of papers at: https://github.com/emrecanacikgoz/awesome-conversational-agents.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 125 of its 140 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

125 of 140 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025Agentic Knowledgeable Self-awareness4 Apr
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems31 Mar
  3. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents3 Mar
  4. 2025Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model12 Feb
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training20 Jan
  6. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuning3 Jan
  7. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents1 Nov
  8. 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
  9. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systems5 Sep
  10. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents21 Jun
  11. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  12. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement17 Jun
12 more
  1. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation17 May
  2. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models19 Mar
  3. 2024Large Multimodal Agents: A Survey23 Feb
  4. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling16 Feb
  5. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents14 Feb
  6. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  7. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  8. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  9. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  11. 2023Cognitive Architectures for Language Agents5 Sep
  12. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 101

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2025s1: Simple test-time scalingcited by 1,311
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  11. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  13. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  14. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  15. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  16. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  17. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  18. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  19. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  20. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  21. 2023Tool Learning with Foundation Modelscited by 302
  22. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  23. 2021Towards Emotional Support Dialog Systemscited by 294
  24. 2023Large Language Models as Tool Makerscited by 293
  25. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  26. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  27. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  28. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  29. 2023Do Large Language Models Know What They Don't Know?cited by 238
  30. 2024Tool Learning with Large Language Models: A Surveycited by 221
  31. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  32. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  33. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  34. 2022TALM: Tool Augmented Language Modelscited by 197
  35. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  36. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  37. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  38. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  2. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems
  3. 2025Conversational Alignment with Artificial Intelligence in Context
  4. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  5. 2025Small Language Models are the Future of Agentic AI
  6. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  7. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems
  8. 2025Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap