MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) model architectures can significantly reduce the number of activated parameters per token, enabling computationally efficient training and inference. However, their large overall parameter counts and model sizes have precluded their widespread usage in resource-constrained settings as all of the parameters must still be loaded into GPU memory. Prior works aim to address this memory bottleneck by offloading certain experts into CPU memory and porting them to GPU memory only when they are activated. In practice, these methods suffer from the significant I/O latency incurred by expert transfer. We present MELINOE, a method that fine-tunes an MoE model to more strongly prefer activating a smaller number of experts per sequence. Caching these preferred experts in GPU memory reduces expert churn and CPU-GPU transfer overhead. MELINOE increases throughput by 1.2-3× over efficient baselines and up to 14.7× over transfer-heavy baselines while retaining or even improving the performance of the model on a downstream task, making it a reliable method for improving MoE inference efficiency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 16 of its 39 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
16 of 39 references.
Closest ancestors · 6
Same direction, newest first
- 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference13 Nov
- 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models14 Jun
- 2025Faster MoE LLM Inference for Extremely Large Models6 May
- 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
- 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
- 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
Other references · 10
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024Qwen2 Technical Reportcited by 2,076
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2024Mixtral of Expertscited by 1,785
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
- 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
- 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
- 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
- 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
- 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
- 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
- 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
- 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
- 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
- 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference