mlsanityData to 14 Oct 2026

Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

arXiv 2110.04725 ↗First posted 10 Oct 2021cs.CL cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

Recent work like GPT-3 has demonstrated excellent performance of Zero-Shot and Few-Shot learning on many natural language processing (NLP) tasks by scaling up model size, dataset size and the amount of computation. However, training a model like GPT-3 requires huge amount of computational resources which makes it challengeable to researchers. In this work, we propose a method that incorporates large-scale distributed training performance into model architecture design. With this method, Yuan 1.0, the current largest singleton language model with 245B parameters, achieves excellent performance on thousands GPUs during training, and the state-of-the-art results on NLP tasks. A data processing method is designed to efficiently filter massive amount of raw data. The current largest high-quality Chinese corpus with 5TB high quality texts is built based on this method. In addition, a calibration and label expansion method is proposed to improve the Zero-Shot and Few-Shot performance, and steady improvement is observed on the accuracy of various tasks. Yuan 1.0 presents strong capacity of natural language generation, and the generated articles are difficult to distinguish from the human-written ones.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
11 more
  1. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  2. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  3. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  4. 2021WebQA: Multihop and Multimodal QAcited by 109
  5. 2021M6: A Chinese Multimodal Pretrainercited by 91
  6. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  7. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  8. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  9. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  10. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  11. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 13; 2024: 7; 2025: 1; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  6. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  8. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  9. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  10. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  11. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  12. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
23 more
  1. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  2. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  3. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  4. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  5. 2022PQLM -- Multilingual Decentralized Portable Quantum Language Model for Privacy Protectioncited by 26
  6. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19
  7. 2022On the Role of Bidirectionality in Language Model Pre-Trainingcited by 15
  8. 2023Ziya2: Data-centric Learning is All LLMs Needcited by 15
  9. 2024TeleChat Technical Reportcited by 15
  10. 2024DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuningcited by 11
  11. 2023YUAN 2.0: A Large Language Model with Localized Filtering-based Attentioncited by 9
  12. 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 8
  13. 2024DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Modelscited by 6
  14. 2023When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenariocited by 5
  15. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  16. 2023Who is leading in AI? An analysis of industry AI researchcited by 4
  17. 2023Oasis: Data Curation and Assessment System for Pretraining of Large Language Modelscited by 3
  18. 2024ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained informationcited by 3
  19. 2024Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challengescited by 2
  20. 2022Changes from Classical Statistics to Modern Statistics and Data Sciencecited by 0
  21. 2023Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sourcescited by 0
  22. 2024LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 0
  23. 2025Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blockscited by 0