SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 51 of its 99 references that are in the map. First posted on arXiv 22 Jul 2026.
Builds on
51 of 99 references.
Closest ancestors · 3
Same direction, newest first
- 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar
- 2026K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model22 Feb
- 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
Other references · 48
- 2017Proximal Policy Optimization Algorithmscited by 14,592
- 2021Training Verifiers to Solve Math Word Problemscited by 9,916
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
- 2017Decoupled Weight Decay Regularizationcited by 1,788
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,307
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 961
36 more
- 2024DeepSeek-V3 Technical Reportcited by 777
- 2025Kimi K2: Open Agentic Intelligencecited by 544
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 542
- 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 515
- 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 325
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 280
- 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 253
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 171
- 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 127
- 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 99
- 2025mHC: Manifold-Constrained Hyper-Connectionscited by 82
- 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 57
- 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 56
- 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 49
- 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 48
- 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 39
- 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 39
- 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 36
- 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 34
- 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 27
- 2025STARK: Strategic Team of Agents for Refining Kernelscited by 24
- 2026AVO: Agentic Variation Operators for Autonomous Evolutionary Searchcited by 22
- 2025OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learningcited by 17
- 2024Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluationcited by 16
- 2024Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMscited by 12
- 2026AscendKernelGen: A Systematic Study of LLM-Based Kernel Generation for Neural Processing Unitscited by 12
- 2025ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modelingcited by 9
- 2026AscendCraft: Automatic Ascend NPU Kernel Generation via DSL-Guided Transcompilationcited by 7
- 2026FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimizationcited by 6
- 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 5
- 2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 5
- 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 5
- 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 3
- 2026AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimizationcited by 3
- 2026Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipelinecited by 2
- 2026Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimizationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
- 2024Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
- 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
- 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
- 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
- 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
- 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
- 2026HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs