mlsanityData to 13 Oct 2026

Training Report of TeleChat3-MoE

arXiv 2512.24157 ↗First posted 30 Dec 2025cs.CLcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 23 of its 34 references that are in the map. First posted on arXiv 30 Dec 2025.

Builds on

23 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Scaling Laws for Neural Language Modelscited by 6,494
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  9. 2017Mixed Precision Trainingcited by 1,147
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
  11. 2020Efficient Transformers: A Surveycited by 785
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
10 more
  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 452
  2. 2015Deep Learning with Limited Numerical Precisioncited by 372
  3. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  4. 2024AI Flow at the Network Edgecited by 44
  5. 2025AI Flow: Perspectives, Scenarios, and Approachescited by 44
  6. 2024Tele-FLM Technical Reportcited by 12
  7. 202452B to 1T: Lessons Learned via Tele-FLM Seriescited by 11
  8. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Modelscited by 10
  9. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  10. 2025TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answeringcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A benchmark for joint dialogue satisfaction, emotion recognition, and emotion state transition predictioncited by 0
  2. 2026Dynamic Knowledge Fusion for Multi-Domain Dialogue State Trackingcited by 0
  3. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  4. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  2. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
  3. 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
  4. 2026Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  6. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  7. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
  8. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing