mlsanityData to 7 Oct 2026

CPM-2: Large-scale Cost-effective Pre-trained Language Models

arXiv 2106.10715 ↗First posted 20 Jun 2021cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

In recent years, the size of pre-trained language models (PLMs) has grown by leaps and bounds. However, efficiency issues of these large-scale PLMs limit their utilization in real-world scenarios. We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. (1) We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. (2) We explore the best practice of prompt tuning with large-scale PLMs. Compared with conventional fine-tuning, prompt tuning significantly reduces the number of task-specific parameters. (3) We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources. Based on our cost-effective pipeline, we pre-train two models: an encoder-decoder bilingual model with 11 billion parameters (CPM-2) and its corresponding MoE version with 198 billion parameters. In our experiments, we compare CPM-2 with mT5 on downstream tasks. Experimental results show that CPM-2 has excellent general language intelligence. Moreover, we validate the efficiency of InfMoE when conducting inference of large-scale models having tens of billions of parameters on a single GPU. All source code and model parameters are available at https://github.com/TsinghuaAI/CPM.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 20 Jun 2021.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
20 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2017Mixed Precision Trainingcited by 1,135
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  9. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  10. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  11. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  12. 2021Fully Hyperbolic Neural Networkscited by 107
  13. 2013Deep Learning of Representations: Looking Forwardcited by 84
  14. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  15. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  16. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  17. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  18. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  19. 2018Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrievalcited by 33
  20. 2021CCPM: A Chinese Classical Poetry Matching Datasetcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 13; 2023: 10; 2024: 9; 2025: 5; 2026: 2.

52 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  7. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  10. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  11. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  12. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
38 more
  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  3. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  4. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  5. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  6. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  7. 2021PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generationcited by 37
  8. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  9. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  10. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  11. 2021More but Correct: Generating Diversified and Entity-revised Medical Responsecited by 31
  12. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  13. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  14. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  15. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  16. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  17. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  18. 2021CUGE: A Chinese Language Understanding and Generation Evaluation Benchmarkcited by 16
  19. 2022LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Modelscited by 16
  20. 2024Can Pre-trained Language Models Understand Chinese Humor?cited by 13
  21. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  22. 2024LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysiscited by 10
  23. 2023LittleMu: Deploying an Online Virtual Teaching Assistant via Heterogeneous Sources Integration and Chain of Teach Promptscited by 9
  24. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7
  25. 2024A Survey of Large Language Models for European Languagescited by 6
  26. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  27. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  28. 2025It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMscited by 5
  29. 2022Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledgecited by 3
  30. 2022Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusterscited by 3
  31. 2024A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generationcited by 3
  32. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Papercited by 2
  33. 2025A Review on Large Language Models for Visual Analyticscited by 1
  34. 2025PCMind-2.1-Kaiyuan-2B Technical Reportcited by 1
  35. 2021FPM: A Collection of Large-scale Foundation Pre-trained Language Modelscited by 0
  36. 2022Few-Shot Natural Language Inference Generation with PDD: Prompt and Dynamic Demonstrationcited by 0
  37. 2023Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sourcescited by 0
  38. 2025FairLangProc: A Python package for fairness in NLPcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  3. 2024A Survey on Efficient Inference for Large Language Models
  4. 2023Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
  5. 2023PolyLM: An Open Source Polyglot Large Language Model
  6. 2024A critical review of methods and challenges in large language models
  7. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Models
  8. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges