# A Blueprint Architecture of Compound AI Systems for Enterprise

arXiv: https://arxiv.org/abs/2406.00584 · HTML: https://mlsanity.com/paper/2406.00584/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)

## Built on by (most cited first, 6 of 6)

- [Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI](https://mlsanity.com/paper/2504.08148/) (2025)
- [Aligning Compound AI Systems via System-level DPO](https://mlsanity.com/paper/2502.17721/) (2025)
- [AIPOM: Agent-aware Interactive Planning for Multi-Agent Systems](https://mlsanity.com/paper/2509.24826/) (2025)
- [Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study](https://mlsanity.com/paper/2604.25724/) (2026)
- [Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layer](https://mlsanity.com/paper/2609.19203/) (2026)
- [BluePixel: Toward Native 4K Agentic Image Editing](https://mlsanity.com/paper/2610.19882/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with ``stream'' serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.