mlsanityData to 14 Oct 2026

Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer

arXiv 2604.00785 ↗First posted 1 Apr 2026cs.LG cs.AI cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Pretraining Large Language Models (LLMs) from scratch requires massive amount of compute. Aurora super computer is an ExaScale machine with 127,488 Intel PVC (Ponte Vechio) GPU tiles. In this work, we showcase LLM pretraining on Aurora at the scale of 1000s of GPU tiles. Towards this effort, we developed Optimus, an inhouse training library with support for standard large model training techniques. Using Optimus, we first pretrained Mula-1B, a 1 Billion dense model and Mula-7B-A1B, a 7 Billion Mixture of Experts (MoE) model from scratch on 3072 GPU tiles for the full 4 trillion tokens of the OLMoE-mix-0924 dataset. We then demonstrated model scaling by pretraining three large MoE models Mula-20B-A2B, Mula-100B-A7B, and Mula-220B-A10B till 100 Billion tokens on the same dataset. On our largest model Mula-220B-A10B, we pushed the compute scaling from 384 to 12288 GPU tiles and observed scaling efficiency of around 90% at 12288 GPU tiles. We significantly improved the runtime performance of MoE models using custom GPU kernels for expert computation, and a novel EP-Aware sharded optimizer resulting in training speedups up to 1.71x. As part of the Optimus library, we also developed a robust set of reliability and fault tolerant features to improve training stability and continuity at scale.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 10 of its 12 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

10 of 12 references.

References · 10

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  10. 2025Aurora: Architecting Argonne's First Exascale Supercomputer for Accelerated Scientific Discoverycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  2. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  3. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
  4. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  5. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  6. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
  7. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference
  8. 2025Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework