mlsanityData to 7 Oct 2026

Towards Pareto Optimal Throughput in Small Language Model Serving

arXiv 2404.03353 ↗First posted 4 Apr 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting that the small memory footprint of SLMs allows for reaching the Pareto-optimal throughput within the resource capacity of a single accelerator. In this regard, we present an initial set of findings demonstrating how model replication can effectively improve resource utilization for serving SLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

17 of 33 references.

References · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023Textbooks Are All You Needcited by 561
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
5 more
  1. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  2. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  3. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  4. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  5. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  2. 2025A Data-driven ML Approach for Maximizing Performance in LLM-Adapter Servingcited by 1
  3. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Servingcited by 0
  4. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
  5. 2026How Should I Pick a Foundation Model for My Robot? In Favor of a Community Evaluation Framework for Social Robotscited by 0