# LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts

arXiv: https://arxiv.org/abs/2601.18089 · HTML: https://mlsanity.com/paper/2601.18089/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 13 of 13)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [REAP the Experts: Why Pruning Prevails for One-Shot MoE compression](https://mlsanity.com/paper/2510.13999/) (2025)
- [Compression Scaling Laws:Unifying Sparsity and Quantization](https://mlsanity.com/paper/2502.16440/) (2025)
- [MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models](https://mlsanity.com/paper/2503.23100/) (2025)
- [Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://mlsanity.com/paper/2506.05508/) (2025)

## Built on by (most cited first, 14 of 14)

- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2604.12374/) (2026)
- [Scalable Training of Mixture-of-Experts Models with Megatron Core](https://mlsanity.com/paper/2603.07685/) (2026)
- [Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism](https://mlsanity.com/paper/2602.04870/) (2026)
- [IsabeLLM: Automated Theorem Proving Applied to Formally Verifying Consensus](https://mlsanity.com/paper/2606.18098/) (2026)
- [Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs](https://mlsanity.com/paper/2607.04371/) (2026)
- [Federation of Experts: Communication Efficient Distributed Inference for Large Language Models](https://mlsanity.com/paper/2605.06206/) (2026)
- [Riemannian Gradient Descent for Low-Rank Architectures](https://mlsanity.com/paper/2606.02328/) (2026)
- [Scaling Adaptive Depth with Norm-Agnostic Residual Networks](https://mlsanity.com/paper/2606.16112/) (2026)
- [Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration](https://mlsanity.com/paper/2608.28511/) (2026)
- [DANTINOX: A Unified Framework for Multi-Paradigm Language Modeling](https://mlsanity.com/paper/2609.17535/) (2026)
- [Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation](https://mlsanity.com/paper/2610.14644/) (2026)
- [LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation](https://mlsanity.com/paper/2610.19975/) (2026)

## Abstract (arXiv)

Mixture of Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal with respect to inference cost, as measured by accuracy per floating-point operation and per parameter. In this work, we revisit MoE design from a hardware-software co-design perspective, grounded in empirical and theoretical considerations. We characterize key performance bottlenecks across diverse deployment regimes, spanning offline high-throughput execution and online, latency-critical inference. Guided by these insights, we introduce LatentMoE, a new model architecture resulting from systematic design exploration and optimized for maximal accuracy per unit of compute. Empirical design space exploration at scales of up to 95B parameters and over a 1T-token training horizon, together with supporting theoretical analysis, shows that LatentMoE consistently outperforms standard MoE architectures in terms of accuracy per FLOP and per parameter. Given its strong performance, the LatentMoE architecture has been adopted by the flagship Nemotron-3 Super and Ultra models and scaled to substantially larger regimes, including longer token horizons and larger model sizes, as reported in Nvidia et al. (arXiv:2512.20856).