# Good things come in small packages: Should we build AI clusters with Lite-GPUs?

arXiv: https://arxiv.org/abs/2501.10187 · HTML: https://mlsanity.com/paper/2501.10187/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 17 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [Reducing Energy Bloat in Large Model Training](https://mlsanity.com/paper/2312.06902/) (2023)
- [Wafer-scale Computing: Advancements, Challenges, and Future Perspectives](https://mlsanity.com/paper/2310.09568/) (2023)
- [Cost-Aware Exploration for Chiplet-Based Architecture with Advanced Packaging Technologies](https://mlsanity.com/paper/2206.07308/) (2022)

## Built on by (most cited first, 1 of 1)

- [RPU -- A Reasoning Processing Unit](https://mlsanity.com/paper/2602.18568/) (2026)

## Abstract (arXiv)

To match the blooming demand of generative AI workloads, GPU designers have so far been trying to pack more and more compute and memory into single complex and expensive packages. However, there is growing uncertainty about the scalability of individual GPUs and thus AI clusters, as state-of-the-art GPUs are already displaying packaging, yield, and cooling limitations. We propose to rethink the design and scaling of AI clusters through efficiently-connected large clusters of Lite-GPUs, GPUs with single, small dies and a fraction of the capabilities of larger GPUs. We think recent advances in co-packaged optics can enable distributing AI workloads onto many Lite-GPUs through high bandwidth and efficient communication. In this paper, we present the key benefits of Lite-GPUs on manufacturing cost, blast radius, yield, and power efficiency; and discuss systems opportunities and challenges around resource, workload, memory, and network management.