mlsanityData to 7 Oct 2026

PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation

arXiv 2109.09519 ↗First posted 20 Sep 2021cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conversations. To train such large models, we adopt the architecture of unified transformer with high computation and parameter efficiency. In addition, we carry out multi-party aware pre-training to better distinguish the characteristic information in social media conversations. With such designs, PLATO-XL successfully achieves superior performances as compared to other approaches in both Chinese and English chitchat. We further explore the capacity of PLATO-XL on other conversational tasks, such as knowledge grounded dialogue and task-oriented conversation. The experimental results indicate that PLATO-XL obtains state-of-the-art results across multiple conversational tasks, verifying its potential as a foundation model of conversational AI.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 20 Sep 2021.

Builds on

39 of 54 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
27 more
  1. 2020Recipes for building an open-domain chatbotcited by 593
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2020The Pushshift Reddit Datasetcited by 289
  5. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  6. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  7. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  8. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  9. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  10. 2018Neural Approaches to Conversational AIcited by 164
  11. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  12. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  13. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  14. 2019Proactive Human-Machine Conversation with Explicit Conversation Goalscited by 87
  15. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  16. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  17. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  18. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  19. 2020Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Accesscited by 58
  20. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  21. 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
  22. 2021EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Trainingcited by 34
  23. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  24. 2017Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNscited by 21
  25. 2021Learning to Select External Knowledge with Multi-Scale Negative Samplingcited by 16
  26. 2019Know More about Each Other: Evolving Dialogue Strategy via Compound Assessmentcited by 8
  27. 2021Dual Slot Selector via Local Reliability Verification for Dialogue State Trackingcited by 5

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 9; 2023: 15; 2024: 7; 2025: 3.

37 papers.

Sort:

By area:

  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  3. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 107
  4. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 50
  5. 2024AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Frameworkcited by 47
  6. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  7. 2023CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Modelscited by 34
  8. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  9. 2024Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximizationcited by 29
  10. 2023Generative Knowledge Selection for Knowledge-Grounded Dialoguescited by 15
  11. 2023ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Humancited by 14
  12. 2021TOD-DA: Towards Boosting the Robustness of Task-oriented Dialogue Modeling on Spoken Conversationscited by 11
25 more
  1. 2022Towards Boosting the Open-Domain Chatbot with Human Feedbackcited by 11
  2. 2022CDConv: A Benchmark for Contradiction Detection in Chinese Conversationscited by 10
  3. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  4. 2022PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Modelcited by 9
  5. 2023GLM-Dialog: Noise-tolerant Pre-training for Knowledge-grounded Dialogue Generationcited by 9
  6. 2023LittleMu: Deploying an Online Virtual Teaching Assistant via Heterogeneous Sources Integration and Chain of Teach Promptscited by 9
  7. 2024Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generationcited by 9
  8. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  9. 2022Link the World: Improving Open-domain Conversation with Dynamic Spatiotemporal-aware Knowledgecited by 6
  10. 2023UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learningcited by 6
  11. 2025Multi-Party Conversational Agents: A Surveycited by 6
  12. 2023Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generationcited by 4
  13. 2024A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generationcited by 3
  14. 2024Exploring the Readiness of Prominent Small Language Models for the Democratization of Financial Literacycited by 3
  15. 2021Analyzing the Limits of Self-Supervision in Handling Bias in Languagecited by 2
  16. 2022PLATO-K: Internal and External Knowledge Enhanced Dialogue Generationcited by 2
  17. 2025Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversationscited by 2
  18. 2021Partner Personas Generation for Diverse Dialogue Generationcited by 1
  19. 2023The economic trade-offs of large language models: A case studycited by 1
  20. 2024A Large Collection of Model-generated Contradictory Responses for Consistency-aware Dialogue Systemscited by 1
  21. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  22. 2023Topic-switch adapted Japanese Dialogue System based on PLATO-2cited by 0
  23. 2023InterviewBot: Real-Time End-to-End Dialogue System to Interview Students for College Admissioncited by 0
  24. 2023Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generationcited by 0
  25. 2024SILC-EFSA: Self-aware In-context Learning Correction for Entity-level Financial Sentiment Analysiscited by 0