mlsanityData to 7 Oct 2026

Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI

arXiv 2504.08148 ↗First posted 10 Apr 2025cs.AI cs.DB cs.DCcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) have gained significant interest in industry due to their impressive capabilities across a wide range of tasks. However, the widespread adoption of LLMs presents several challenges, such as integration into existing applications and infrastructure, utilization of company proprietary data, models, and APIs, and meeting cost, quality, responsiveness, and other requirements. To address these challenges, there is a notable shift from monolithic models to compound AI systems, with the premise of more powerful, versatile, and reliable applications. However, progress thus far has been piecemeal, with proposals for agentic workflows, programming models, and extended LLM capabilities, without a clear vision of an overall architecture. In this paper, we propose a 'blueprint architecture' for compound AI systems for orchestrating agents and data for enterprise applications. In our proposed architecture the key orchestration concept is 'streams' to coordinate the flow of data and instructions among agents. Existing proprietary models and APIs in the enterprise are mapped to 'agents', defined in an 'agent registry' that serves agent metadata and learned representations for search and planning. Agents can utilize proprietary data through a 'data registry' that similarly registers enterprise data of various modalities. Tying it all together, data and task 'planners' break down, map, and optimize tasks and queries for given quality of service (QoS) requirements such as cost, accuracy, and latency. We illustrate an implementation of the architecture for a use-case in the HR domain and discuss opportunities and challenges for 'agentic AI' in the enterprise.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 51 of its 61 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

51 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024BMW Agents -- A Framework For Task Automation Through Multi-Agent Collaboration28 Jun
  2. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents5 Mar
  3. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System23 Feb
  4. 2024Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions2 Feb
  5. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  6. 2023AutoAgents: A Framework for Automatic Agent Generation29 Sep
  7. 2023Agents: An Open-source Framework for Autonomous Language Agents14 Sep
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  9. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug

Other references · 42

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
30 more
  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  5. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  6. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  7. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  8. 2023Large Language Models as Tool Makerscited by 293
  9. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  10. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  11. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  12. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  13. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  14. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  15. 2023An LLM Compiler for Parallel Function Callingcited by 170
  16. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  17. 2023A Survey on Fairness in Large Language Modelscited by 91
  18. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  19. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  20. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  21. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 47
  22. 2023Exploring Large Language Model for Graph Data Understanding in Online Job Recommendationscited by 45
  23. 2024A Declarative System for Optimizing AI Workloadscited by 35
  24. 2023Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Modelscited by 31
  25. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  26. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  27. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  28. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  29. 2024A Blueprint Architecture of Compound AI Systems for Enterprisecited by 5
  30. 2024CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  3. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  4. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuningcited by 3
  5. 2025Technical Implementation of Tippy: Multi-Agent Architecture and System Design for Drug Discovery Laboratory Automationcited by 2
  6. 2025An Agentic AI for a New Paradigm in Business Process Developmentcited by 2
  7. 2026Blue Data Intelligence Layer: Streaming Data and Agents for Multi-source Multi-modal Data-Centric Applicationscited by 1
  8. 2026ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestrationcited by 1
  9. 2026Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions
  2. 2025Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystems
  3. 2026Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study
  4. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
  5. 2026Agentic Nesting: A New Methodology for Existing Enterprise Application Integration and Services
  6. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems
  7. 2026Towards Trustworthy and Cost-Efficient Data Integration: From Na"ive RAG to Agentic RAG
  8. 2026Orchestral AI: A Framework for Agent Orchestration