# Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI arXiv: https://arxiv.org/abs/2504.08148 ยท HTML: https://mlsanity.com/paper/2504.08148/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. - Builds on 51 of its 61 references that are in the map. - First posted on arXiv 10 Apr 2025. ## Placed in - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 30 of 51) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022) - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021) - [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022) - [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023) - [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023) - [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023) - [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023) - [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023) - [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023) - [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023) - [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021) - [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023) - [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023) - [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023) - [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023) - [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024) - [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023) - [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023) - [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024) ## Built on by (most cited first, 9 of 9) - [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025) - [Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards](https://mlsanity.com/paper/2507.03041/) (2025) - [Routine: A Structural Planning Framework for LLM Agent System in Enterprise](https://mlsanity.com/paper/2507.14447/) (2025) - [TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuning](https://mlsanity.com/paper/2602.08306/) (2026) - [Technical Implementation of Tippy: Multi-Agent Architecture and System Design for Drug Discovery Laboratory Automation](https://mlsanity.com/paper/2507.17852/) (2025) - [An Agentic AI for a New Paradigm in Business Process Development](https://mlsanity.com/paper/2507.21823/) (2025) - [ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestration](https://mlsanity.com/paper/2605.20485/) (2026) - [Blue Data Intelligence Layer: Streaming Data and Agents for Multi-source Multi-modal Data-Centric Applications](https://mlsanity.com/paper/2604.15233/) (2026) - [Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows](https://mlsanity.com/paper/2607.00828/) (2026) ## Abstract (arXiv) Large language models (LLMs) have gained significant interest in industry due to their impressive capabilities across a wide range of tasks. However, the widespread adoption of LLMs presents several challenges, such as integration into existing applications and infrastructure, utilization of company proprietary data, models, and APIs, and meeting cost, quality, responsiveness, and other requirements. To address these challenges, there is a notable shift from monolithic models to compound AI systems, with the premise of more powerful, versatile, and reliable applications. However, progress thus far has been piecemeal, with proposals for agentic workflows, programming models, and extended LLM capabilities, without a clear vision of an overall architecture. In this paper, we propose a 'blueprint architecture' for compound AI systems for orchestrating agents and data for enterprise applications. In our proposed architecture the key orchestration concept is 'streams' to coordinate the flow of data and instructions among agents. Existing proprietary models and APIs in the enterprise are mapped to 'agents', defined in an 'agent registry' that serves agent metadata and learned representations for search and planning. Agents can utilize proprietary data through a 'data registry' that similarly registers enterprise data of various modalities. Tying it all together, data and task 'planners' break down, map, and optimize tasks and queries for given quality of service (QoS) requirements such as cost, accuracy, and latency. We illustrate an implementation of the architecture for a use-case in the HR domain and discuss opportunities and challenges for 'agentic AI' in the enterprise.