mlsanityData to 14 Oct 2026

HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices

arXiv 2605.05819 ↗First posted 7 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, they often suffer from substantial accuracy degradation or severe throughput bottlenecks. Recent error compensation methods recover accuracy through auxiliary LoRA-style branches, and we observe that these branches are inherently amenable to offloading: they require substantial parameter storage but access only a small subset of compensation parameters during each inference step. Motivated by this opportunity, we propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU, and further introduces an asynchronous compensation pipeline and sensitivity-aware dynamic rank allocation to hide compensation overhead and maximize accuracy recovery. Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks compared to compression model and sustaining a maximum speedup of 10.4x compared to full-precision model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

25 of 35 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
13 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  2. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  3. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  4. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  5. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  7. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  8. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 51
  9. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  10. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Modelscited by 17
  11. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Modelscited by 9
  12. 2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8
  13. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7

Built on by

Not cited yet.