mlsanityData to 7 Oct 2026

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

arXiv 2608.28511 ↗First posted 28 Aug 2026cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3% fewer GPU-hours while improving average downstream score and inference throughput.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 28 of its 30 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

28 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free10 May
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
12 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  2. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  3. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  4. 2025Pretraining Large Language Models with NVFP4cited by 67
  5. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  6. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  7. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  8. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
  9. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  10. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  11. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 12
  12. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  2. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  3. 2025Advancing Expert Specialization for Better MoE
  4. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  5. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training
  6. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  7. 2025Mixture of Experts in Large Language Models
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Experts