AI Compute Architecture and Evolution Trends
Not on the map. Why?
Abstract (arXiv)
The focus of AI development has shifted from academic research to practical applications. However, AI development faces numerous challenges at various levels. This article will attempt to analyze the opportunities and challenges of AI from several different perspectives using a structured approach. This article proposes a seven-layer model for AI compute architecture, including Physical Layer, Link Layer, Neural Network Layer, Context Layer, Agent Layer, Orchestrator Layer, and Application Layer, from bottom to top. It also explains the three stages in the evolution of large language models (LLMs) using the proposed 7-layer model. For each layer, we describe the development trajectory and key technologies. In Layers 1 and 2 we discuss AI computing issues and the impact of Scale-Up and Scale-Out strategies on computing architecture. In Layer 3 we explore two different development paths for LLMs. In Layer 4 we discuss the impact of contextual memory on LLMs and compares it to traditional processor memory. In Layers 5 to 7 we discuss the trends of AI agents and explore the issues in evolution from a single AI agent to an AI-based ecosystem, and their impact on the AI industry.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 73 of its 179 references that are in the map. First posted on arXiv 29 Aug 2025.
Builds on
73 of 179 references.
References · 73
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
38 more
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
- 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2020What is the State of Neural Network Pruning?cited by 583
- 2022Efficiently Scaling Transformer Inferencecited by 539
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
- 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
- 2023Learning to Compress Prompts with Gist Tokenscited by 390
- 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
- 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
- 2023A Survey on Model Compression for Large Language Modelscited by 269
- 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
- 2021Accelerating Sparse Deep Neural Networkscited by 223
- 2023Large Language Model Guided Tree-of-Thoughtcited by 221
- 2025Survey on Evaluation of LLM-based Agentscited by 204
- 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
- 2022FP8 Formats for Deep Learningcited by 196
- 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
- 2024An Introduction to Vision-Language Modelingcited by 178
- 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
- 2023Microscaling Data Formats for Deep Learningcited by 152
- 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
- 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
- 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
- 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
- 2019Fast Sparse ConvNetscited by 86
- 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
- 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
- 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
- 2024Tiny Machine Learning: Progress and Futurescited by 40
- 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
- 2024Differentiable Weightless Neural Networkscited by 28
50 most cited of 73 All 73 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.