mlsanityData to 14 Oct 2026

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

arXiv 2608.30076 ↗First posted 30 Aug 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 31 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

22 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
5 more
  1. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  2. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  3. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  4. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  5. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  2. 2026Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs
  3. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  4. 2026Fast NF4 Dequantization Kernels for Large Language Model Inference
  5. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression