mlsanityData to 7 Oct 2026

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

arXiv 2303.02868 ↗First posted 6 Mar 2023cs.LG cs.DCcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as WeChat, QQ, and Tencent Advertisement, have been opted in to gain the power of pre-trained models. In this work, we present Angel-PTM, a productive deep learning system designed for pre-training and fine-tuning Transformer models. Angel-PTM can train extremely large-scale models with hierarchical memory efficiently. The key designs of Angel-PTM are the fine-grained memory management via the Page abstraction and a unified scheduling method that coordinate the computations, data movements, and communications. Furthermore, Angel-PTM supports extreme model scaling with SSD storage and implements the lock-free updating mechanism to address the SSD I/O bandwidth bottlenecks. Experimental results demonstrate that Angel-PTM outperforms existing systems by up to 114.8% in terms of maximum model scale as well as up to 88.9% in terms of training throughput. Additionally, experiments on GPT3-175B and T5-MoE-1.2T models utilizing hundreds of GPUs verify the strong scalability of Angel-PTM.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 35 of its 69 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

35 of 69 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2022Training language models to follow instructions with human feedbackcited by 15,712
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
23 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  13. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  14. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  15. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  16. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  17. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  18. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  19. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  20. 2022Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updatescited by 15
  21. 2022OSDP: Optimal Sharded Data Parallel for Distributed Deep Learningcited by 12
  22. 2019Efficient Memory Management for GPU-based Deep Learning Systemscited by 6
  23. 2022A Review of Machine Learning-based Failure Management in Optical Networkscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 3.

8 papers.

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  3. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 28
  4. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  5. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashingcited by 5
  6. 2024VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMscited by 3
  7. 2025Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storagecited by 3
  8. 2025From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guaranteescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Ultra-Sparse Memory Network
  2. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  3. 2026SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
  4. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  5. 2024Zamba: A Compact 7B SSM Hybrid Model
  6. 2023LEMON: Lossless model expansion
  7. 2026Learning Rate Transfer for Hybrid Transformer-SSM Architectures
  8. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling