mlsanityData to 14 Oct 2026

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

arXiv 2408.11743 ↗First posted 21 Aug 2024cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

As inference on Large Language Models (LLMs) emerges as an important workload in machine learning applications, weight quantization has become a standard technique for efficient GPU deployment. Quantization not only reduces model size, but has also been shown to yield substantial speedups for single-user inference, due to reduced memory movement, with low accuracy impact. Yet, it remains open whether speedups are achievable also in batched settings with multiple parallel clients, which are highly relevant for practical serving. It is unclear whether GPU kernels can be designed to remain practically memory-bound, while supporting the substantially increased compute requirements of batched workloads. This paper resolves this question positively by describing the design of Mixed-precision Auto-Regressive LINear kernels, called MARLIN. Concretely, given a model whose weights are compressed via quantization to, e.g., 4 bits per element, MARLIN shows that batchsizes up to 16-32 can be supported with close to maximum (4×) quantization speedup, and larger batchsizes up to 64-128 with gradually decreasing, but still significant, acceleration. MARLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining, and bespoke quantization support. Our experiments show that MARLIN's near-optimal performance on individual LLM layers across different scenarios can also lead to end-to-end LLM inference speedups (of up to 2.8×) when integrated with the popular vLLM serving engine. Finally, MARLIN is extensible to further compression techniques, like NVIDIA 2:4 sparsity, leading to additional speedups.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

19 of 32 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
7 more
  1. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  2. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  3. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  4. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  5. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 23
  6. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  7. 2022Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviorscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 30.

52 papers.

Sort:

By area:

  1. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  2. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  3. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 16
  4. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  5. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  6. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
  7. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  8. 2025Rectified Sparse Attentioncited by 10
  9. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cachecited by 8
  10. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  11. 2024CursorCore: Assist Programming through Aligning Anythingcited by 7
  12. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
38 more
  1. 2026Statistically-Lossless Quantization of Large Language Modelscited by 6
  2. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  3. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  4. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  5. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  6. 2025DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theorycited by 3
  7. 2026FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Designcited by 3
  8. 2026W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUscited by 3
  9. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Modelscited by 3
  10. 2025Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programscited by 2
  11. 2025PLaMo 2 Technical Reportcited by 2
  12. 2025iServe: An Intent-based Serving System for LLMscited by 1
  13. 2026A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPUcited by 1
  14. 2026RPU -- A Reasoning Processing Unitcited by 1
  15. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  16. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 1
  17. 2026Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inferencecited by 1
  18. 2026Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantizationcited by 1
  19. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generationcited by 1
  20. 2026QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstructioncited by 1
  21. 2026Quantizing Looped Transformers: Feedback Exposure and Calibration Blindnesscited by 1
  22. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 0
  23. 2025Fair-GPTQ: Bias-Aware Quantization for Large Language Modelscited by 0
  24. 2025PerfDojo: Automated ML Library Generation for Heterogeneous Architecturescited by 0
  25. 2026On the Quantization Robustness of Diffusion Language Models in Coding Benchmarkscited by 0
  26. 2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compressioncited by 0
  27. 2026BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantizationcited by 0
  28. 2026LLM Compression with Jointly Optimizing Architectural and Quantization choicescited by 0
  29. 2026AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesiscited by 0
  30. 2026SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Servingcited by 0
  31. 2026Route-Block Membership Selects Packed-AWQ Arithmetic: A Controlled Single-Fixture Mechanism Studycited by 0
  32. 2026CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weightscited by 0
  33. 2026FlashDrive: Flash Vision-Language-Action Inference for Autonomous Drivingcited by 0
  34. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  35. 2026Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compressioncited by 0
  36. 2026Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compressioncited by 0
  37. 2026Structured Transforms for Low-Overhead Quantization of Language Modelscited by 0
  38. 2026Disaggregated Quantization: Specializing LLM Prefill and Decodecited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  2. 2026Fast NF4 Dequantization Kernels for Large Language Model Inference
  3. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  4. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
  5. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
  6. 2023SqueezeLLM: Dense-and-Sparse Quantization
  7. 2025Enabling Dynamic Sparsity in Quantized LLM Inference
  8. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference