mlsanityData to 14 Oct 2026

LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts

arXiv 2601.18089 ↗First posted 26 Jan 2026cs.LG cs.AIcited by 14

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal with respect to inference cost, as measured by accuracy per floating-point operation and per parameter. In this work, we revisit MoE design from a hardware-software co-design perspective, grounded in empirical and theoretical considerations. We characterize key performance bottlenecks across diverse deployment regimes, spanning offline high-throughput execution and online, latency-critical inference. Guided by these insights, we introduce LatentMoE, a new model architecture resulting from systematic design exploration and optimized for maximal accuracy per unit of compute. Empirical design space exploration at scales of up to 95B parameters and over a 1T-token training horizon, together with supporting theoretical analysis, shows that LatentMoE consistently outperforms standard MoE architectures in terms of accuracy per FLOP and per parameter. Given its strong performance, the LatentMoE architecture has been adopted by the flagship Nemotron-3 Super and Ultra models and scaled to substantially larger regimes, including longer token horizons and larger model sizes, as reported in Nvidia et al. (arXiv:2512.20856).

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 13 of its 17 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

13 of 17 references.

References · 13

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  4. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  6. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  7. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  9. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  10. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  11. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 11
  12. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Modelscited by 10
1 more
  1. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Kimi K3: Open Frontier Intelligencecited by 138
  2. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  3. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  4. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  5. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2
  6. 2026IsabeLLM: Automated Theorem Proving Applied to Formally Verifying Consensuscited by 1
  7. 2026Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMscited by 1
  8. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Modelscited by 0
  9. 2026Riemannian Gradient Descent for Low-Rank Architecturescited by 0
  10. 2026Scaling Adaptive Depth with Norm-Agnostic Residual Networkscited by 0
  11. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0
  12. 2026DANTINOX: A Unified Framework for Multi-Paradigm Language Modelingcited by 0
2 more
  1. 2026Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocationcited by 0
  2. 2026LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  4. 2026MobileMoE: Scaling On-Device Mixture of Experts
  5. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
  6. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
  7. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource