mlsanityData to 14 Oct 2026

Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon

arXiv 2604.16957 ↗First posted 18 Apr 2026cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon, enabling 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac -- a configuration impossible with all existing inference frameworks. Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation via custom Metal compute shaders, eliminating all intermediate dequantization matrices. Across 330 experiments spanning two model families (Gemma 4 31B and Llama 3.1 70B), the fused sdpa_int4 kernel achieves 48x attention speedup at 128K context over the dequantize-then-attend baseline, reduces KV cache memory from 40 GB to 12.5 GB (3.2x compression), and maintains identical top-1 token predictions to FP16 inference. We further provide the first cross-architecture analysis of KV cache quantization methods, revealing that the attention scale factor -- not model size -- determines whether angular quantization schemes like PolarQuant succeed or fail, with Gemma 4's attn_scale=1.0 amplifying directional error 25-100x more than Llama's standard 1/sqrt(d) scaling.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 17 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

13 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  8. 2018Online normalizer calculation for softmaxcited by 162
  9. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  10. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  11. 2025PolarQuant: Quantizing KV Caches with Polar Transformationcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026JustFit: Just-in-Time State Management for Local LLM Servingcited by 0
  2. 2026SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktopscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  2. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  3. 2026Runtime-Certified Bounded-Error Quantized Attention
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  5. 2026PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection
  6. 2026AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
  7. 2026BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
  8. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference