mlsanityData to 14 Oct 2026

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

arXiv 2606.08761 ↗First posted 7 Jun 2026cs.DC cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio (ρ) as the primary hardware indicator: the W4A4-g128 kernel yields 2.0--2.5× speedup on RTX~3090 (ρ=16) yet degrades to 0.43--0.47× on A100 (ρ=64) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build APEX4, which co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0%--4.4% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to 1.66× end-to-end speedup on L40S (ρ=8), and 1.78× on RTX~3090 (ρ=16), 2.09× on A40 (ρ=16), while recovering A100 (ρ=64) to 1.20--1.40× via the mixed-granularity mode. Our code is available at https://github.com/APEX4-W4A4/APEX4-W4A4.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 7 Jun 2026.

Builds on

22 of 28 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
10 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  6. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 24
  7. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  8. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Modelscited by 16
  9. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Servingcited by 8
  10. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache
  2. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
  3. 2026Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance
  4. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  5. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  6. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
  7. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  8. 2026The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference