mlsanityData to 14 Oct 2026

Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

arXiv 2609.38090 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-constrained, single-GPU systems. However, this benefit is difficult to realize because expert parameters dominate memory, and token-level routing is dynamic, unpredictable, and skewed. Prior work using offloading and caching remains fundamentally reactive, as systems wait for router outputs before moving experts, leading to inefficient cache utilization and an inability to overlap transfers with compute under tight VRAM budgets. To address these challenges, we propose Mira, an algorithm-system co-design that enables high-capacity MoE inference on a single GPU. Mira shifts from a reactive to a proactive stance by coupling predictive expert management with a tailored quantization format. It introduces lightweight per-layer predictors that anticipate expert usage two layers ahead, enabling proactive prefetching. These predictions feed a two-tier HOT+STAGE GPU cache managed by token-level routing telemetry to retain frequently used experts while staging predicted ones. To minimize transfer overhead, Mira implements a custom compression for expert parameters, which reduces metadata and improves packing efficiency, while minimally degrading accuracy. Mira is implemented as a fully integrated runtime that coordinates predictors, caching policies, and quantized transfers to maximize overlap between communication and compute. Our experiments show that Mira reduces expert-induced stalls. Compared against state-of-the-art baselines, Mira achieves a 5.71x speedup in average throughput on a memory-constrained GPU. It accelerates Time-to-First-Token by 11.71x and achieves a 3.84x average speedup in beam search inference, demonstrating its effectiveness across diverse inference scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 34 of its 48 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

34 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU9 May
  2. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  3. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  5. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  6. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2024Mixtral of Expertscited by 1,823
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
14 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  2. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  3. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  4. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  5. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  6. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  7. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  8. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  9. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  10. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  11. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  12. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 23
  13. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  14. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  2. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  3. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  4. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  5. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  6. 2025Accelerating MoE Model Inference with Expert Sharding
  7. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  8. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference