Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
Not on the map. Why?
Abstract (arXiv)
Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number. We present an attribution study on four NVIDIA RTX A5000 GPUs, 24 GiB each, on a single host with NVLink-bridged pairs. A matched intermediate stack that keeps the faster runtime without the quantized kernel splits the full speedup into a runtime part and a kernel and quantization part. Under matched greedy decoding the full stack reaches 2.58× end to end, with the runtime change accounting for about two thirds of that gain on a logarithmic scale; across three similar model families the kernel and quantization part moves by at most 1.5%. Sharding one instance across all four cards falls well below doubling: a profiler trace attributes about 80% of the per token shortfall to coordination, and an NVLink versus PCIe control on the same hardware shows similar realized bandwidth on both links, pointing away from link bandwidth as the cause. Whether to run one sharded instance or several independent ones depends on the workload and the model, with the ranking reversing on the larger model: the smaller model splits between sharding and multiple instances by workload, while the larger model favors two paired instances on every workload. Quantization extends sustainable concurrent users roughly four times past a reproducible half precision memory cliff. Differences in sampling mode and prompt pool between the two stacks are documented as threats to validity.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
17 of 28 references.
References · 17
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
5 more
- 2022Efficiently Scaling Transformer Inferencecited by 529
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2019MLPerf Inference Benchmarkcited by 195
- 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
- 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
Built on by
Citing papers per year: 2026: 1.
1 paper.