mlsanityData to 14 Oct 2026

DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment

arXiv 2508.06041 ↗First posted 8 Aug 2025cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

How can we effectively handle queries for on-device large language models (LLMs) with varying runtime constraints, such as latency and accuracy? Multi-scale quantization addresses this challenge by enabling memory-efficient runtime model adaptation of LLMs through the overlaying of multiple model variants quantized to different bitwidths. Meanwhile, an important question still remains open-ended: how can models be properly configured to match a target precision or latency? While mixed-precision offers a promising solution, we take this further by leveraging the key observation that the sensitivity of each layer dynamically changes across decoding steps. Building on this insight, we introduce DP-LLM, a novel mechanism that dynamically assigns precision to each layer based on input values. Experimental results across multiple models and benchmarks demonstrate that DP-LLM achieves a superior performance-latency trade-off, outperforming prior approaches.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

31 of 46 references.

References · 31

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
19 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  8. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  12. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  13. 2017Adaptive Quantization for Deep Neural Networkcited by 67
  14. 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 64
  15. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  16. 2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
  17. 2020ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobilescited by 15
  18. 2025Matryoshka Quantizationcited by 14
  19. 2025On Evaluating Performance of LLM Inference Serving Systemscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 9
  2. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 2
  3. 2026River-LLM: Large Language Model Seamless Exit Based on KV Sharecited by 1
  4. 2026Pricing the Risk of Runtime Compression: Anytime-Valid Admission and a Served-Output Law for Compressed Serving Statecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inference
  2. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
  3. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  4. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  5. 2023Efficient LLM Inference on CPUs
  6. 2024One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments
  7. 2026QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
  8. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs