mlsanityData to 7 Oct 2026

Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

arXiv 2402.10517 ↗First posted 16 Feb 2024cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

Recently, considerable efforts have been directed towards compressing Large Language Models (LLMs), which showcase groundbreaking capabilities across diverse applications but entail significant deployment costs due to their large sizes. Meanwhile, much less attention has been given to mitigating the costs associated with deploying multiple LLMs of varying sizes despite its practical significance. Thus, this paper introduces any-precision LLM, extending the concept of any-precision DNN to LLMs. Addressing challenges in any-precision LLM, we propose a lightweight method for any-precision quantization of LLMs, leveraging a post-training quantization framework, and develop a specialized software engine for its efficient serving. As a result, our solution significantly reduces the high costs of deploying multiple, different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such as 3, 4, ..., n bits, into a memory footprint comparable to a single n-bit LLM. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput, proving itself to be a compelling option for deployment of multiple, different-sized LLMs. Our code is open-sourced and available online.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

24 of 41 references.

References · 24

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
12 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  4. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  7. 2023Speculative Decoding with Big Little Decodercited by 163
  8. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  9. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  10. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  11. 2019Any-Precision Deep Neural Networkscited by 39
  12. 2019ALERT: Accurate Learning for Energy and Timelinesscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 24; 2026: 17.

50 papers.

Sort:

By area:

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  2. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 28
  3. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  4. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 17
  5. 2025ORANSight-2.0: Foundational LLMs for O-RANcited by 13
  6. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  7. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 9
  8. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 8
  9. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  10. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
  11. 2025RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarmscited by 7
  12. 2025RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remappingcited by 6
38 more
  1. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  2. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Designcited by 5
  3. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantizationcited by 5
  4. 2024CDQuant: Greedy Coordinate Descent for Accurate LLM Quantizationcited by 4
  5. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4
  6. 2025FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devicescited by 4
  7. 2025DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignmentcited by 4
  8. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  9. 2026STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Trainingcited by 4
  10. 2025Theoretical Foundations of Scaling Law in Familial Modelscited by 3
  11. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMscited by 3
  12. 2026Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMscited by 3
  13. 2024Log-Time K-Means Clustering for 1D Data: Novel Approaches with Proof and Implementationcited by 2
  14. 2025HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrievalcited by 2
  15. 2025LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformerscited by 2
  16. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Modelscited by 2
  17. 2024Scaling Laws For Mixed Quantizationcited by 1
  18. 2025ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuningcited by 1
  19. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  20. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inferencecited by 1
  21. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1
  22. 2026MixQuant: Adaptive Mixed-Precision Quantization for Large Language Modelscited by 1
  23. 2025TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scalingcited by 0
  24. 2025ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMscited by 0
  25. 2025OTARo: Once Tuning for All Precisions toward Robust On-Device LLMscited by 0
  26. 2025Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challengescited by 0
  27. 2025Atom: Efficient On-Device Video-Language Pipelines Through Modular Reusecited by 0
  28. 2026QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switchingcited by 0
  29. 2026MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLMcited by 0
  30. 2026TMPDiff: Temporal Mixed-Precision for Diffusion Modelscited by 0
  31. 2026M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verificationcited by 0
  32. 2026ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Servingcited by 0
  33. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  34. 2026Multi-Bitwidth Quantization for LLMs Using Additive Codebookscited by 0
  35. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generationcited by 0
  36. 2026Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repaircited by 0
  37. 2026Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Cachescited by 0
  38. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memorycited by 0