mlsanityData to 14 Oct 2026

Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design

arXiv 2511.17127 ↗First posted 21 Nov 2025cs.CL cs.AI cs.DCcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts over Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE, available at https://huggingface.co/Zyphra/ZAYA1-base) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 54 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

29 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Muon is Scalable for LLM Training24 Feb
  2. 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters22 Jul
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 26

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
14 more
  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  3. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  5. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 53
  6. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  7. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  8. 2025Mid-Training of Large Language Models: A Surveycited by 21
  9. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  10. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 14
  11. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  12. 2024The Case for Co-Designing Model Architectures with Hardwarecited by 11
  13. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  14. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  2. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 6
  3. 2026ZAYA1-8B Technical Reportcited by 4
  4. 2026ZAYA1-VL-8B Technical Reportcited by 2
  5. 2026Scaling Adaptive Depth with Norm-Agnostic Residual Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
  2. 2026Mixture-of-Kittens: MoE Megakernel for NVL72s
  3. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
  4. 2026AMDKernelVault: Large-Scale Datasets and Agentic Training for AMD GPU Kernel Optimization
  5. 2025MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
  6. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap
  7. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  8. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging