mlsanityData to 7 Oct 2026

PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

arXiv 2303.10845 ↗First posted 20 Mar 2023cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-Sigma. With parameter inherent from PanGu-α, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Sigma provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 67 of its 73 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

67 of 73 references.

References · 67

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  14. 2018Hierarchical Neural Story Generationcited by 1,182
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  17. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  18. 2021Measuring Coding Challenge Competence With APPScited by 879
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  20. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  21. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  22. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  23. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  24. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  25. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  26. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  27. 2021Hash Layers For Large Sparse Modelscited by 236
  28. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  29. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  30. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  31. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  32. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  33. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  34. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  35. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  36. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  37. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  38. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 23; 2025: 9; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  8. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  9. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  10. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  11. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  12. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
38 more
  1. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  2. 2024A Survey on Transformer Compressioncited by 44
  3. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  4. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  5. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  7. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  8. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  9. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  10. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22
  11. 2024SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understandingcited by 20
  12. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  13. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  14. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  15. 2023Prompt-Based Length Controlled Generation with Reinforcement Learningcited by 14
  16. 2024PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Modelscited by 14
  17. 2024LogLM: From Task-based to Instruction-based Automated Log Analysiscited by 13
  18. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  19. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 10
  20. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  21. 2023Large Language Models for Mathematicianscited by 7
  22. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  23. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  24. 2024Trillion Parameter AI Serving Infrastructure for Scientific Discovery: A Survey and Visioncited by 5
  25. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  26. 2023Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Modelscited by 4
  27. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  28. 2024FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUscited by 3
  29. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Expertscited by 3
  30. 2023Should ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Eracited by 1
  31. 2023The Promise and Peril of Artificial Intelligence -- Violet Teaming Offers a Balanced Path Forwardcited by 1
  32. 2024OpenCarbonEval: A Unified Carbon Emission Estimation Framework in Large-Scale AI Modelscited by 1
  33. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  34. 2025Prior Distribution and Model Confidencecited by 1
  35. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0
  36. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inferencecited by 0
  37. 2025FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translationcited by 0
  38. 2026StackingNet: Collective Inference Across Independent AI Foundation Modelscited by 0

50 most cited of 55 All 55 (JSON)