mlsanityData to 7 Oct 2026

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

arXiv 2608.08081 ↗First posted 8 Aug 2026cs.NE cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh--Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring O(d log d) operations and 256 stored parameters per head versus O(d2) and 16,384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state---a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16,GB budget and Nemotron-H 120B within 32,GB, running interactively at 9--19 tok/s with near-zero perplexity degradation (DeltaPPL ≤ +0.0012) and 100% retrieval accuracy at 32K context.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 16 of its 26 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

16 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
  2. 2025CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
  3. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  4. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  5. 2026ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits
  6. 2026MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
  7. 2025Collaborative Compression for Large-Scale MoE Deployment on Edge
  8. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation