DevNous: An LLM-Based Multi-Agent System for Grounding IT Project Management in Unstructured Conversation
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
The manual translation of unstructured team dialogue into the structured artifacts required for Information Technology (IT) project governance is a critical bottleneck in modern information systems management. We introduce DevNous, a Large Language Model-based (LLM) multi-agent expert system, to automate this unstructured-to-structured translation process. DevNous integrates directly into team chat environments, identifying actionable intents from informal dialogue and managing stateful, multi-turn workflows for core administrative tasks like automated task formalization and progress summary synthesis. To quantitatively evaluate the system, we introduce a new benchmark of 160 realistic, interactive conversational turns. The dataset was manually annotated with a multi-label ground truth and is publicly available. On this benchmark, DevNous achieves an exact match turn accuracy of 81.3% and a multiset F1-Score of 0.845, providing strong evidence for its viability. The primary contributions of this work are twofold: (1) a validated architectural pattern for developing ambient administrative agents, and (2) the introduction of the first robust empirical baseline and public benchmark dataset for this challenging problem domain.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 24 of its 80 references that are in the map. First posted on arXiv 12 Aug 2025.
Builds on
24 of 80 references.
Closest ancestors · 3
Same direction, newest first
- 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective4 Jun
- 2023A Survey on Large Language Model based Autonomous Agents22 Aug
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
Other references · 21
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
9 more
- 2023ChatDev: Communicative Agents for Software Developmentcited by 919
- 2023Benchmarking Large Language Models for News Summarizationcited by 445
- 2023Tool Learning with Foundation Modelscited by 302
- 2024AI Agents That Mattercited by 219
- 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
- 2024Efficacy of Synthetic Data as a Benchmarkcited by 30
- 2024Experimenting a New Programming Practice with LLMscited by 17
- 2024Exploring Human-AI Collaboration in Agile: Customised LLM Meeting Assistantscited by 9
- 2025The Effectiveness of Large Language Models in Transforming Unstructured Text to Standardized Formatscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Pragmos: A Process Agentic Modeling System
- 2024Adaptive In-conversation Team Building for Language Model Agents
- 2026ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
- 2025T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning
- 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
- 2024MARCO: Multi-Agent Real-time Chat Orchestration
- 2025Cognitive Agents Powered by Large Language Models for Agile Software Project Management
- 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions