mlsanityData to 14 Oct 2026

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

arXiv 2609.34077 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems also fits: Expert routing strategies

Abstract (arXiv)

Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loads them on demand, so decoding speed depends on how many experts each token must fetch. Caching and prefetching reduce this cost only as far as the routing allows. Router-only fine-tuning can reshape the routing to reuse experts, but it keeps the experts frozen, so they cannot adapt to the tokens the new routing sends them. We propose MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss alone. During fine-tuning, a learnable binary mask restricts the Top-K routing of each layer to a subset of experts, and the experts adapt to the tokens redirected to them. At inference, the learned mask becomes a soft prior that re-ranks experts, so every expert remains selectable. We simulate a GPU cache of 4 experts per layer for Mixtral-8x7B and 12 for DeepSeek-V2-Lite. MaskCoFT cuts expert fetches per token by 23.7% and 10.1% relative to the base model. In real offloading system serving, it lowers the time per output token by up to 16.4% and 5.5%, respectively. Its average accuracy over nine benchmarks stays above the base model by 0.92 and 0.53 points.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 32 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

25 of 32 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference26 May
  2. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE14 May
  3. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems4 Dec
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  5. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  6. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling23 Oct
  8. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
  9. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  10. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  11. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2024Mixtral of Expertscited by 1,823
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  8. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  9. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  10. 2018MaskConnect: Connectivity Learning by Gradient Descentcited by 9
  11. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  12. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 3
1 more
  1. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating MoE Model Inference with Expert Sharding
  2. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  3. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  4. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  5. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  6. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  7. 2026MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
  8. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving