# EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training

arXiv: https://arxiv.org/abs/2108.01547 · HTML: https://mlsanity.com/paper/2108.01547/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://mlsanity.com/paper/2006.16779/) (2020)
- [A Large-Scale Chinese Short-Text Conversation Dataset](https://mlsanity.com/paper/2008.03946/) (2020)
- [Personalized Dialogue Generation with Diversified Traits](https://mlsanity.com/paper/1901.09672/) (2019)
- [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://mlsanity.com/paper/2012.00413/) (2020)
- [Pchatbot: A Large-Scale Dataset for Personalized Chatbot](https://mlsanity.com/paper/2009.13284/) (2020)

## Built on by (most cited first, 30 of 34)

- [Safety Assessment of Chinese Large Language Models](https://mlsanity.com/paper/2304.10436/) (2023)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [COLD: A Benchmark for Chinese Offensive Language Detection](https://mlsanity.com/paper/2201.06025/) (2022)
- [PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing](https://mlsanity.com/paper/2303.10845/) (2023)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation](https://mlsanity.com/paper/2109.09519/) (2021)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- [CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models](https://mlsanity.com/paper/2311.16832/) (2023)
- [EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training](https://mlsanity.com/paper/2203.09313/) (2022)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models](https://mlsanity.com/paper/2305.11262/) (2023)
- [Rethinking and Refining the Distinct Metric](https://mlsanity.com/paper/2202.13587/) (2022)
- [CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI](https://mlsanity.com/paper/2205.14727/) (2022)
- [ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human](https://mlsanity.com/paper/2304.07849/) (2023)
- [Towards Boosting the Open-Domain Chatbot with Human Feedback](https://mlsanity.com/paper/2208.14165/) (2022)
- [Estimating Soft Labels for Out-of-Domain Intent Detection](https://mlsanity.com/paper/2211.05561/) (2022)
- [CDConv: A Benchmark for Contradiction Detection in Chinese Conversations](https://mlsanity.com/paper/2210.08511/) (2022)
- [A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment](https://mlsanity.com/paper/2305.08200/) (2023)
- [Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation](https://mlsanity.com/paper/2406.07850/) (2024)
- [PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model](https://mlsanity.com/paper/2203.17090/) (2022)
- [Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue](https://mlsanity.com/paper/2210.07783/) (2022)
- [Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation](https://mlsanity.com/paper/2212.01810/) (2022)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning](https://mlsanity.com/paper/2306.10543/) (2023)
- [A Review and Roadmap of Deep Learning Causal Discovery in Different Variable Paradigms](https://mlsanity.com/paper/2209.06367/) (2022)
- [Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generation](https://mlsanity.com/paper/2305.07393/) (2023)
- [A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generation](https://mlsanity.com/paper/2404.03491/) (2024)
- [PLATO-K: Internal and External Knowledge Enhanced Dialogue Generation](https://mlsanity.com/paper/2211.00910/) (2022)
- [Semi-Supervised Lifelong Language Learning](https://mlsanity.com/paper/2211.13050/) (2022)
- [A Principle-Driven Adaptive Policy for Group Cognitive Stimulation Dialogue for Elderly with Cognitive Impairment](https://mlsanity.com/paper/2603.10034/) (2026)

## Abstract (arXiv)

Although pre-trained language models have remarkably enhanced the generation ability of dialogue systems, open-domain Chinese dialogue systems are still limited by the dialogue data and the model size compared with English ones. In this paper, we propose EVA, a Chinese dialogue system that contains the largest Chinese pre-trained dialogue model with 2.8B parameters. To build this model, we collect the largest Chinese dialogue dataset named WDC-Dialogue from various public social media. This dataset contains 1.4B context-response pairs and is used as the pre-training corpus of EVA. Extensive experiments on automatic and human evaluation show that EVA outperforms other Chinese pre-trained dialogue models especially in the multi-turn interaction of human-bot conversations.