mlsanityData to 7 Oct 2026

CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

arXiv 2109.05729 ↗First posted 13 Sep 2021cs.CLcited by 81

Not on the map. Why?

Abstract (arXiv)

In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

39 of 46 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
27 more
  1. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  2. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  3. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  4. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  5. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  6. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  7. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  8. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  9. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  10. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  11. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  12. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  13. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  14. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  15. 2020FLAT: Chinese NER Using Flat-Lattice Transformercited by 67
  16. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  17. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  18. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  19. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  20. 2018DRCD: a Chinese Machine Reading Comprehension Datasetcited by 57
  21. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  22. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  23. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42
  24. 2019NEZHA: Neural Contextualized Representation for Chinese Language Understandingcited by 39
  25. 2019ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representationscited by 35
  26. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
  27. 2020AMBERT: A Pre-trained Language Model with Multi-Grained Tokenizationcited by 22

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 28; 2023: 27; 2024: 21; 2025: 4.

81 papers.

Sort:

By area:

  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  3. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  4. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 90
  5. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  6. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 53
  7. 2022D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chatcited by 37
  8. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  9. 2022MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correctioncited by 33
  10. 2023LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluationcited by 33
  11. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  12. 2022A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systemscited by 32
38 more
  1. 2022SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parsercited by 32
  2. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  3. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  4. 2023LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streamingcited by 27
  5. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  6. 2023A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluationcited by 24
  7. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  8. 2022LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphscited by 15
  9. 2022JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understandingcited by 15
  10. 2023Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialoguecited by 15
  11. 2022FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correctioncited by 13
  12. 2023Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correctioncited by 13
  13. 2024Can Pre-trained Language Models Understand Chinese Humor?cited by 13
  14. 2023Causal reasoning in typical computer vision taskscited by 12
  15. 2022Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learningcited by 11
  16. 2022GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentationcited by 11
  17. 2022Fine-grained Contrastive Learning for Definition Generationcited by 10
  18. 2023Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modellingcited by 10
  19. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  20. 2022Clues Before Answers: Generation-Enhanced Multiple-Choice QAcited by 9
  21. 2023NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Textscited by 9
  22. 2024Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correctioncited by 9
  23. 2022Nominal Metaphor Generation with Multitask Learningcited by 8
  24. 2023Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluationcited by 7
  25. 2022CC-Riddle: A Question Answering Dataset of Chinese Character Riddlescited by 6
  26. 2022Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialoguescited by 6
  27. 2023UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learningcited by 6
  28. 2023Enhancing Dialogue Summarization with Topic-Aware Global- and Local- Level Centralitycited by 5
  29. 2024Alirector: Alignment-Enhanced Chinese Grammatical Error Correctorcited by 5
  30. 2022SEE-Few: Seed, Expand and Entail for Few-shot Named Entity Recognitioncited by 4
  31. 2022SkillNet-NLG: General-Purpose Natural Language Generation with a Sparsely Activated Approachcited by 3
  32. 2022Focus Is What You Need For Chinese Grammatical Error Correctioncited by 3
  33. 2022TextBox 2.0: A Text Generation Library with Pre-trained Language Modelscited by 3
  34. 2023CSED: A Chinese Semantic Error Diagnosis Corpuscited by 3
  35. 2023ArcGPT: A Large Language Model Tailored for Real-world Archival Applicationscited by 3
  36. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  37. 2024Inconsistent dialogue responses and how to recover from themcited by 3
  38. 2024Towards Reliable and Empathetic Depression-Diagnosis-Oriented Chatscited by 3

50 most cited of 81 All 81 (JSON)