mlsanityData to 7 Oct 2026

Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?

arXiv 2106.01561 ↗First posted 3 Jun 2021cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when question answering.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

21 of 30 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
9 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  3. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  4. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  5. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  6. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  7. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  8. 2019Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanationcited by 43
  9. 2020SemEval-2020 Task 4: Commonsense Validation and Explanationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 19; 2023: 21; 2024: 8; 2025: 8; 2026: 3.

62 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  5. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  6. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  7. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  8. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  9. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  10. 2022Recitation-Augmented Language Modelscited by 65
  11. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  12. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
38 more
  1. 2023Evaluating Open-QA Evaluationcited by 35
  2. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  3. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  4. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  5. 2022ProQA: Structural Prompt-based Pre-training for Unified Question Answeringcited by 21
  6. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17
  7. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 15
  8. 2023How Well Do Large Language Models Truly Ground?cited by 14
  9. 2023Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awarenesscited by 14
  10. 2022Towards Fine-grained Causal Reasoning and QAcited by 13
  11. 2022Robot Task Planning and Situation Handling in Open Worldscited by 10
  12. 2022Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversationcited by 10
  13. 2024Beyond Memorization: The Challenge of Random Memory Access in Language Modelscited by 8
  14. 2022A Unified Encoder-Decoder Framework with Entity Memorycited by 7
  15. 2023Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Modelscited by 7
  16. 2024Knowledge Verification to Nip Hallucination in the Budcited by 7
  17. 2022Closed-book Question Generation via Contrastive Learningcited by 6
  18. 2023Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answeringcited by 6
  19. 2023Learning to Generalize for Cross-domain QAcited by 6
  20. 2022A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Modelscited by 5
  21. 2022OPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experiencecited by 5
  22. 2023KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generationcited by 5
  23. 2023KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphscited by 5
  24. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adaptercited by 5
  25. 2025GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledgecited by 5
  26. 2022Pre-training Language Models with Deterministic Factual Knowledgecited by 4
  27. 2023RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answeringcited by 4
  28. 2023Post Turing: Mapping the landscape of LLM Evaluationcited by 4
  29. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
  30. 2022Towards Answering Open-ended Ethical Quandary Questionscited by 3
  31. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  32. 2023Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Modelscited by 3
  33. 2024Identifying the Achilles' Heel: An Iterative Method for Dynamically Uncovering Factual Errors in Large Language Modelscited by 3
  34. 2025Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Trainingcited by 3
  35. 2026MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmarkcited by 3
  36. 2022Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Basecited by 2
  37. 2022WikiOmnia: generative QA corpus on the whole Russian Wikipediacited by 2
  38. 2022Artificial Intelligence and Natural Language Processing and Understanding in Space: A Methodological Framework and Four ESA Case Studiescited by 2

50 most cited of 62 All 62 (JSON)