mlsanityData to 14 Oct 2026

FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

arXiv 2603.13364 ↗First posted 9 Mar 2026cs.CV cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 45 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

37 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2024Mixtral of Expertscited by 1,823
24 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  10. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  11. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  12. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  13. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  14. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 59
  15. 2025LongCat-Flash Technical Reportcited by 58
  16. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  17. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  18. 2024Multi-Head Mixture-of-Expertscited by 31
  19. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 24
  20. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
  21. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  22. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  23. 2024Llama 3 Meets MoE: Efficient Upcyclingcited by 6
  24. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
  2. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
  3. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
  4. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
  5. 2026Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE
  6. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  8. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference