mlsanityData to 14 Oct 2026

Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM

arXiv 2601.14327 ↗First posted 20 Jan 2026cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert Pruning (LAEP) algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert pruning approaches that operate primarily in the post-training phase, the proposed algorithm enhances training efficiency by selectively pruning underutilized experts and reorganizing experts across computing devices according to token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this algorithm delivers a 49% boost in pre-training efficiency and a 33.3% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 27 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

23 of 27 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression15 Oct
  2. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Models31 Jul
  3. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models10 Apr
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  5. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  7. 2024Mixtral of Experts8 Jan
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 15

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  12. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
3 more
  1. 2023YUAN 2.0: A Large Language Model with Localized Filtering-based Attentioncited by 9
  2. 2025MMTABREAL: Real-World Benchmark for Multimodal Table Understandingcited by 9
  3. 2026Yuan3.0 Flash: An Open Multimodal Large Language Model for Enterprise Applicationscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
  2. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  3. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  4. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference
  5. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  6. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  7. 2025Advancing Expert Specialization for Better MoE
  8. 2025MoE Pathfinder: Trajectory-driven Expert Pruning