mlsanityData to 14 Oct 2026

MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

arXiv 2604.21026 ↗First posted 22 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

25 of 36 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2024SpinQuant: LLM quantization with learned rotationscited by 471
13 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  3. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  9. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  10. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  11. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  12. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  13. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9

Built on by

Not cited yet.