# Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation

arXiv: https://arxiv.org/abs/2211.01587 · HTML: https://mlsanity.com/paper/2211.01587/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding](https://mlsanity.com/paper/2104.08455/) (2021)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)

## Built on by (most cited first, 10 of 10)

- [Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators](https://mlsanity.com/paper/2310.07289/) (2023)
- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph](https://mlsanity.com/paper/2503.15990/) (2025)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models](https://mlsanity.com/paper/2305.14775/) (2023)
- [Knowledge Rumination for Pre-trained Language Models](https://mlsanity.com/paper/2305.08732/) (2023)
- [Improving Factuality for Dialogue Response Generation via Graph-Based Knowledge Augmentation](https://mlsanity.com/paper/2506.12496/) (2025)
- [FineDialFact: A benchmark for Fine-grained Dialogue Fact Verification](https://mlsanity.com/paper/2508.05782/) (2025)
- [Internal and External Knowledge Interactive Refinement Framework for Knowledge-Intensive Question Answering](https://mlsanity.com/paper/2408.12979/) (2024)

## Abstract (arXiv)

Recent advances in large-scale pre-training provide large models with the potential to learn knowledge from the raw text. It is thus natural to ask whether it is possible to leverage these large models as knowledge bases for downstream tasks. In this work, we answer the aforementioned question in unsupervised knowledge-grounded conversation. We explore various methods that best elicit knowledge from large models. Our human study indicates that, though hallucinations exist, large models post the unique advantage of being able to output common sense and summarize facts that cannot be directly retrieved from the search engine. To better exploit such generated knowledge in dialogue generation, we treat the generated knowledge as a noisy knowledge source and propose the posterior-based reweighing as well as the noisy training strategy. Empirical results on two benchmarks show advantages over the state-of-the-art methods.