mlsanityData to 7 Oct 2026

PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model

arXiv 2203.17090 ↗First posted 31 Mar 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive amount of dialogue data from scratch, we aim to build a powerful dialogue model with relatively fewer data and computation costs by inheriting valuable language capabilities and knowledge from PLMs. To this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been proven well-performed on a variety of Chinese natural language tasks. We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety. We show that PanGu-Bot outperforms state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA (Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects. We also demonstrate that PanGu-Bot can be easily deployed to generate emotional responses without further training. Throughout our empirical analysis, we also point out that the PanGu-Bot response quality, knowledge correctness, and safety are still far from perfect, and further explorations are indispensable to building reliable and smart dialogue systems. Our model and code will be available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/PanGu-Bot soon.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

51 of 60 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  12. 2022Red Teaming Language Models with Language Modelscited by 983
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  4. 2020Recipes for building an open-domain chatbotcited by 593
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  7. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  8. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  9. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  10. 2021Internet-Augmented Dialogue Generationcited by 217
  11. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  12. 2018Gender Bias in Neural Natural Language Processingcited by 210
  13. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  14. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  15. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  16. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  17. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  18. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  19. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  20. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  21. 2019Does Gender Matter? Towards Fairness in Dialogue Systemscited by 89
  22. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  23. 2019Proactive Human-Machine Conversation with Explicit Conversation Goalscited by 87
  24. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  25. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  26. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  27. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  28. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  29. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  30. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  31. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  32. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  33. 2021PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generationcited by 37
  34. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  35. 2021EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Trainingcited by 34
  36. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  37. 2020Multi-Modal Open-Domain Dialoguecited by 19
  38. 2021UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialoguescited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 5; 2024: 1.

9 papers.

  1. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  2. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  3. 2023ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Humancited by 14
  4. 2024Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Modelscited by 10
  5. 2023This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Modelscited by 9
  6. 2022Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variablescited by 5
  7. 2022KPT: Keyword-guided Pre-training for Grounded Dialog Generationcited by 3
  8. 2023ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialoguecited by 3
  9. 2022Modeling Complex Dialogue Mappings via Sentence Semantic Segmentation Guided Conditional Variational Auto-Encodercited by 1