# Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

arXiv: https://arxiv.org/abs/2402.10517 · HTML: https://mlsanity.com/paper/2402.10517/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 24 of 24)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models](https://mlsanity.com/paper/2306.02272/) (2023)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization](https://mlsanity.com/paper/2305.14152/) (2023)
- [Any-Precision Deep Neural Networks](https://mlsanity.com/paper/1911.07346/) (2019)
- [ALERT: Accurate Learning for Energy and Timeliness](https://mlsanity.com/paper/1911.00119/) (2019)

## Built on by (most cited first, 30 of 54)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://mlsanity.com/paper/2505.07004/) (2025)
- [QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache](https://mlsanity.com/paper/2502.10424/) (2025)
- [Progressive Mixed-Precision Decoding for Efficient LLM Inference](https://mlsanity.com/paper/2410.13461/) (2024)
- [ORANSight-2.0: Foundational LLMs for O-RAN](https://mlsanity.com/paper/2503.05200/) (2025)
- [AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://mlsanity.com/paper/2411.10606/) (2024)
- [Adaptive Draft-Verification for Efficient Large Language Model Decoding](https://mlsanity.com/paper/2407.12021/) (2024)
- [AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs](https://mlsanity.com/paper/2510.10467/) (2025)
- [Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment](https://mlsanity.com/paper/2509.20214/) (2025)
- [DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization](https://mlsanity.com/paper/2412.20185/) (2024)
- [RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping](https://mlsanity.com/paper/2501.04052/) (2025)
- [RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarms](https://mlsanity.com/paper/2507.01378/) (2025)
- [UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs](https://mlsanity.com/paper/2512.03383/) (2025)
- [Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design](https://mlsanity.com/paper/2503.18869/) (2025)
- [FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization](https://mlsanity.com/paper/2506.12024/) (2025)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization](https://mlsanity.com/paper/2406.17542/) (2024)
- [AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization](https://mlsanity.com/paper/2409.16546/) (2024)
- [FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices](https://mlsanity.com/paper/2501.07139/) (2025)
- [DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment](https://mlsanity.com/paper/2508.06041/) (2025)
- [STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training](https://mlsanity.com/paper/2604.06836/) (2026)
- [SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference](https://mlsanity.com/paper/2512.12990/) (2025)
- [Theoretical Foundations of Scaling Law in Familial Models](https://mlsanity.com/paper/2512.23407/) (2025)
- [MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models](https://mlsanity.com/paper/2602.03537/) (2026)
- [ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs](https://mlsanity.com/paper/2602.17698/) (2026)
- [Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs](https://mlsanity.com/paper/2604.07888/) (2026)
- [Scaling Laws For Mixed Quantization](https://mlsanity.com/paper/2410.06722/) (2024)
- [Log-Time K-Means Clustering for 1D Data: Novel Approaches with Proof and Implementation](https://mlsanity.com/paper/2412.15295/) (2024)
- [HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval](https://mlsanity.com/paper/2506.07296/) (2025)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)

## Abstract (arXiv)

Recently, considerable efforts have been directed towards compressing Large Language Models (LLMs), which showcase groundbreaking capabilities across diverse applications but entail significant deployment costs due to their large sizes. Meanwhile, much less attention has been given to mitigating the costs associated with deploying multiple LLMs of varying sizes despite its practical significance. Thus, this paper introduces any-precision LLM, extending the concept of any-precision DNN to LLMs. Addressing challenges in any-precision LLM, we propose a lightweight method for any-precision quantization of LLMs, leveraging a post-training quantization framework, and develop a specialized software engine for its efficient serving. As a result, our solution significantly reduces the high costs of deploying multiple, different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such as 3, 4, ..., n bits, into a memory footprint comparable to a single n-bit LLM. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput, proving itself to be a compelling option for deployment of multiple, different-sized LLMs. Our code is open-sourced and available online.