# MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models arXiv: https://arxiv.org/abs/2408.11743 · HTML: https://mlsanity.com/paper/2408.11743/ ## Key facts - Cited by 52 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 32 references that are in the map. - First posted on arXiv 21 Aug 2024. ## Builds on (most cited first, 19 of 19) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022) - [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022) - [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023) - [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022) - [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024) - [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023) - [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023) - [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023) - [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022) - [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024) - [Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production](https://mlsanity.com/paper/2211.10017/) (2022) - [Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU](https://mlsanity.com/paper/2301.03598/) (2023) - [Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward](https://mlsanity.com/paper/2402.01799/) (2024) - [Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors](https://mlsanity.com/paper/2206.02874/) (2022) ## Built on by (most cited first, 30 of 52) - [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024) - [InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation](https://mlsanity.com/paper/2509.24663/) (2025) - [Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models](https://mlsanity.com/paper/2503.22879/) (2025) - [QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs](https://mlsanity.com/paper/2510.11696/) (2025) - [MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE](https://mlsanity.com/paper/2507.00390/) (2025) - [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025) - [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2505.11415/) (2025) - [Rectified Sparse Attention](https://mlsanity.com/paper/2506.04108/) (2025) - [BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache](https://mlsanity.com/paper/2503.18773/) (2025) - [SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models](https://mlsanity.com/paper/2508.03351/) (2025) - [CursorCore: Assist Programming through Aligning Anything](https://mlsanity.com/paper/2410.07002/) (2024) - [UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs](https://mlsanity.com/paper/2512.03383/) (2025) - [Statistically-Lossless Quantization of Large Language Models](https://mlsanity.com/paper/2605.02404/) (2026) - [EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts](https://mlsanity.com/paper/2606.18967/) (2026) - [BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference](https://mlsanity.com/paper/2511.10054/) (2025) - [A Proximal Operator for Inducing 2:4-Sparsity](https://mlsanity.com/paper/2501.18015/) (2025) - [LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs](https://mlsanity.com/paper/2511.06174/) (2025) - [DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theory](https://mlsanity.com/paper/2501.06417/) (2025) - [FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Design](https://mlsanity.com/paper/2601.15710/) (2026) - [W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs](https://mlsanity.com/paper/2601.16536/) (2026) - [MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models](https://mlsanity.com/paper/2602.03537/) (2026) - [Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programs](https://mlsanity.com/paper/2504.16214/) (2025) - [PLaMo 2 Technical Report](https://mlsanity.com/paper/2509.04897/) (2025) - [iServe: An Intent-based Serving System for LLMs](https://mlsanity.com/paper/2501.13111/) (2025) - [A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU](https://mlsanity.com/paper/2602.17693/) (2026) - [RPU -- A Reasoning Processing Unit](https://mlsanity.com/paper/2602.18568/) (2026) - [TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly](https://mlsanity.com/paper/2603.19296/) (2026) - [Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate](https://mlsanity.com/paper/2604.13806/) (2026) - [Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference](https://mlsanity.com/paper/2605.13915/) (2026) - [Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization](https://mlsanity.com/paper/2605.16423/) (2026) ## Abstract (arXiv) As inference on Large Language Models (LLMs) emerges as an important workload in machine learning applications, weight quantization has become a standard technique for efficient GPU deployment. Quantization not only reduces model size, but has also been shown to yield substantial speedups for single-user inference, due to reduced memory movement, with low accuracy impact. Yet, it remains open whether speedups are achievable also in batched settings with multiple parallel clients, which are highly relevant for practical serving. It is unclear whether GPU kernels can be designed to remain practically memory-bound, while supporting the substantially increased compute requirements of batched workloads. This paper resolves this question positively by describing the design of Mixed-precision Auto-Regressive LINear kernels, called MARLIN. Concretely, given a model whose weights are compressed via quantization to, e.g., 4 bits per element, MARLIN shows that batchsizes up to 16-32 can be supported with close to maximum (4×) quantization speedup, and larger batchsizes up to 64-128 with gradually decreasing, but still significant, acceleration. MARLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining, and bespoke quantization support. Our experiments show that MARLIN's near-optimal performance on individual LLM layers across different scenarios can also lead to end-to-end LLM inference speedups (of up to 2.8×) when integrated with the popular vLLM serving engine. Finally, MARLIN is extensible to further compression techniques, like NVIDIA 2:4 sparsity, leading to additional speedups.