mlsanityData to 14 Oct 2026

LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind

arXiv 2508.15601 ↗First posted 21 Aug 2025cs.DC cs.PFcited by 10

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Mixed-precision inference techniques reduce the memory and computational demands of Large Language Models (LLMs) by applying hybrid precision formats to model weights, activations, and KV caches. However, existing systems struggle to (i) automatically generalize across diverse hardware architectures and precision formats, often requiring fragmented, hand-tuned kernels, and (ii) fully exploit available memory and compute resources, often causing performance bottlenecks. To address these problems, we propose TurboMind, a generalizable and efficient mixed-precision LLM inference engine of LMDeploy. TurboMind is built around two hardware-aware mixed-precision pipelines: A General Matrix Multiply (GEMM) pipeline that optimizes matrix operations through offline weight packing and online acceleration, and an attention pipeline that enables efficient attention computation with different Query, Key, and Value precision combinations. These pipelines are enabled by four key techniques: (i) Hardware-aware weight packing and (ii) adaptive head alignment for generalizability, and (iii) instruction-level parallelism and (iv) a KV memory loading pipeline for efficiency. We conduct comprehensive evaluations of LMDeploy powered by TurboMind across sixteen popular LLMs and four representative GPU architectures. Results demonstrate that LMDeploy achieves up to 61% lower serving latency (30% on average) and up to 156% higher throughput (58% on average) in mixed-precision workloads compared to existing mixed-precision frameworks, establishing consistent performance improvements across all tested configurations and hardware types. This work is open-sourced and publicly available at https://github.com/InternLM/lmdeploy.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 67 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

30 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
14 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  6. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
  7. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  8. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  9. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  10. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  11. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  12. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 24
  13. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inferencecited by 6
  14. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMscited by 14
  2. 2025GTM: Simulating the World of Tools for AI Agentscited by 14
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  4. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  5. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  6. 2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
  7. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
  8. 2026RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworkscited by 2
  9. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  10. 2026Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient LLM Inference on CPUs
  2. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  3. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  4. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
  5. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  6. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  7. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  8. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inference