# APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

arXiv: https://arxiv.org/abs/2606.08761 · HTML: https://mlsanity.com/paper/2606.08761/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [Benchmarking and Dissecting the Nvidia Hopper GPU Architecture](https://mlsanity.com/paper/2402.13499/) (2024)
- [Scaling Law for Quantization-Aware Training](https://mlsanity.com/paper/2505.14302/) (2025)
- [QQQ: Quality Quattuor-Bit Quantization for Large Language Models](https://mlsanity.com/paper/2406.09904/) (2024)
- [LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving](https://mlsanity.com/paper/2509.01229/) (2025)
- [COMET: Towards Partical W4A4KV4 LLMs Serving](https://mlsanity.com/paper/2410.12168/) (2024)

## Built on by (most cited first, 1 of 1)

- [Transforms for LLM Quantization: The Great Inversion and Format Co-Design](https://mlsanity.com/paper/2608.25188/) (2026)

## Abstract (arXiv)

W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio (ρ) as the primary hardware indicator: the W4A4-g128 kernel yields 2.0--2.5× speedup on RTX~3090 (ρ=16) yet degrades to 0.43--0.47× on A100 (ρ=64) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build APEX4, which co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0%--4.4% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to 1.66× end-to-end speedup on L40S (ρ=8), and 1.78× on RTX~3090 (ρ=16), 2.09× on A40 (ρ=16), while recovering A100 (ρ=64) to 1.20--1.40× via the mixed-granularity mode. Our code is available at https://github.com/APEX4-W4A4/APEX4-W4A4.