mlsanityData to 7 Oct 2026

Good things come in small packages: Should we build AI clusters with Lite-GPUs?

arXiv 2501.10187 ↗First posted 17 Jan 2025cs.AR cs.AI cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

To match the blooming demand of generative AI workloads, GPU designers have so far been trying to pack more and more compute and memory into single complex and expensive packages. However, there is growing uncertainty about the scalability of individual GPUs and thus AI clusters, as state-of-the-art GPUs are already displaying packaging, yield, and cooling limitations. We propose to rethink the design and scaling of AI clusters through efficiently-connected large clusters of Lite-GPUs, GPUs with single, small dies and a fraction of the capabilities of larger GPUs. We think recent advances in co-packaged optics can enable distributing AI workloads onto many Lite-GPUs through high bandwidth and efficient communication. In this paper, we present the key benefits of Lite-GPUs on manufacturing cost, blast radius, yield, and power efficiency; and discuss systems opportunities and challenges around resource, workload, memory, and network management.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 61 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

22 of 61 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2024Large Language Models: A Surveycited by 506
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
10 more
  1. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  3. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  4. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  5. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  6. 2024Fast On-device LLM Inference with NPUscited by 58
  7. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  8. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  9. 2023Wafer-scale Computing: Advancements, Challenges, and Future Perspectivescited by 6
  10. 2022Cost-Aware Exploration for Chiplet-Based Architecture with Advanced Packaging Technologiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RPU -- A Reasoning Processing Unitcited by 1