mlsanityData to 14 Oct 2026

FORGE: Procedural Weight Generation for Sparse MoE Inference

arXiv 2610.16890 ↗First posted 12 Oct 2026cs.AR cs.DC cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We introduce FORGE (Fast On-chip Rotor Generation for Experts), a procedural weight generation system for Sparse Mixture-of-Experts (MoE) inference. Rather than storing and loading every expert weight tensor from HBM, we show how a compact accelerator inside each Streaming Multiprocessor can regenerate them on-the-fly from O(log2 d) rotor parameters per projection. Per replaced layer, gigabytes of stored expert weights are approximated with under a megabyte of seeds. This addresses the dual memory bottleneck of autoregressive Sparse MoE decode, where per-expert FFN operations are HBM-bandwidth-bound at small routed-token counts and the aggregate expert footprint forces multi-node Expert Parallelism (EP) with all-to-all traffic. We realize the approach as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution. With 30% of the MoE FFN layers replaced across five Sparse MoE models, we estimate 1.3-2.8x higher system throughput at matched GPU count and 1.10-1.15x lower per-user decode latency at short context. In exchange, downstream accuracy on three model families falls below that of the same model fine-tuned without rotor replacement, and the replacement ratio controls this tradeoff. FORGE also expands KV cache capacity by 1.2-2.6x. It is agnostic to the bit width at which weights are stored, so it can compose with quantization. The RGU is fully implemented/verified in SystemVerilog at ~0.9% SM area overhead (0.032 mm2/SM, 7nm).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 40 of its 71 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

40 of 71 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 35

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
23 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  3. 2016HyperNetworkscited by 1,027
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  13. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  14. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  15. 2021Accelerating Sparse Deep Neural Networkscited by 223
  16. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  17. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  18. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  19. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 76
  20. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 14
  21. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
  22. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 11
  23. 2025Composing Linear Layers from Irreduciblescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
  2. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  3. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
  4. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  5. 2025Accelerating MoE Model Inference with Expert Sharding
  6. 2021M6-T: Exploring Sparse Expert Models and Beyond
  7. 2022Accelerating Distributed MoE Training and Inference with Lina
  8. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts