mlsanityData to 14 Oct 2026

FloE: On-the-Fly MoE Inference on Memory-constrained GPU

arXiv 2505.05950 ↗First posted 9 May 2025cs.LGcited by 9

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3x compression of parameters per expert in Mixtral-8x7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090 - all with only a 4.4% - 7.6% average performance degradation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 63 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

28 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  2. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  3. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  4. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  5. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  6. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2024Mixtral of Expertscited by 1,823
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  11. 2024Phi-4 Technical Reportcited by 711
  12. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
10 more
  1. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  3. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  4. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  5. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
  6. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  7. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  8. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  9. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  10. 2024CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsificationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 20
  2. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  3. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  4. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  5. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  6. 2025VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reusecited by 1
  7. 2026MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Modelscited by 0
  8. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0
  9. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
  3. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models
  4. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  5. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  6. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  7. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  8. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference