# Prompting as Probing: Using Language Models for Knowledge Base Construction

arXiv: https://arxiv.org/abs/2208.11057 · HTML: https://mlsanity.com/paper/2208.11057/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [BERTese: Learning to Speak to BERT](https://mlsanity.com/paper/2103.05327/) (2021)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Inductive Entity Representations from Text via Link Prediction](https://mlsanity.com/paper/2010.03496/) (2020)
- [Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2110.08173/) (2021)
- [An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models](https://mlsanity.com/paper/2109.02772/) (2021)
- [Approximate Knowledge Graph Query Answering: From Ranking to Binary Classification](https://mlsanity.com/paper/2102.11389/) (2021)

## Built on by (most cited first, 30 of 32)

- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text](https://mlsanity.com/paper/2308.02357/) (2023)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://mlsanity.com/paper/2305.16635/) (2023)
- [I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token](https://mlsanity.com/paper/2412.06676/) (2024)
- [Towards Ontology Construction with Language Models](https://mlsanity.com/paper/2309.09898/) (2023)
- [Large Scale Knowledge Washing](https://mlsanity.com/paper/2405.16720/) (2024)
- [Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecture](https://mlsanity.com/paper/2502.04558/) (2025)
- [Rethinking Language Models as Symbolic Knowledge Graphs](https://mlsanity.com/paper/2308.13676/) (2023)
- [Knowledge Engineering using Large Language Models](https://mlsanity.com/paper/2310.00637/) (2023)
- [Evaluating the Knowledge Base Completion Potential of GPT](https://mlsanity.com/paper/2310.14771/) (2023)
- [Knowledge Base Construction for Knowledge-Augmented Text-to-SQL](https://mlsanity.com/paper/2505.22096/) (2025)
- [Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Prediction](https://mlsanity.com/paper/2209.08966/) (2022)
- [Completeness, Recall, and Negation in Open-World Knowledge Bases: A Survey](https://mlsanity.com/paper/2305.05403/) (2023)
- [Task-specific Pre-training and Prompt Decomposition for Knowledge Graph Population with Language Models](https://mlsanity.com/paper/2208.12539/) (2022)
- [Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization](https://mlsanity.com/paper/2310.12794/) (2023)
- [Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata](https://mlsanity.com/paper/2309.08491/) (2023)
- [LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Models](https://mlsanity.com/paper/2308.13207/) (2023)
- [Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation](https://mlsanity.com/paper/2403.05523/) (2024)
- [Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities](https://mlsanity.com/paper/2508.01290/) (2025)
- [Extracting Multi-valued Relations from Language Models](https://mlsanity.com/paper/2307.03122/) (2023)
- [InFact: Informativeness Alignment for Improved LLM Factuality](https://mlsanity.com/paper/2505.20487/) (2025)
- [Textual Entailment for Effective Triple Validation in Object Prediction](https://mlsanity.com/paper/2401.16293/) (2024)
- [Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning](https://mlsanity.com/paper/2408.17070/) (2024)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)
- [From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering](https://mlsanity.com/paper/2412.17701/) (2024)
- [Pretrained LLMs Learn Multiple Types of Uncertainty](https://mlsanity.com/paper/2505.21218/) (2025)
- [Evaluating Class Membership Relations in Knowledge Graphs using Large Language Models](https://mlsanity.com/paper/2404.17000/) (2024)

## Abstract (arXiv)

Language Models (LMs) have proven to be useful in various downstream applications, such as summarisation, translation, question answering and text classification. LMs are becoming increasingly important tools in Artificial Intelligence, because of the vast quantity of information they can store. In this work, we present ProP (Prompting as Probing), which utilizes GPT-3, a large Language Model originally proposed by OpenAI in 2020, to perform the task of Knowledge Base Construction (KBC). ProP implements a multi-step approach that combines a variety of prompting techniques to achieve this. Our results show that manual prompt curation is essential, that the LM must be encouraged to give answer sets of variable lengths, in particular including empty answer sets, that true/false questions are a useful device to increase precision on suggestions generated by the LM, that the size of the LM is a crucial factor, and that a dictionary of entity aliases improves the LM score. Our evaluation study indicates that these proposed techniques can substantially enhance the quality of the final predictions: ProP won track 2 of the LM-KBC competition, outperforming the baseline by 36.4 percentage points. Our implementation is available on https://github.com/HEmile/iswc-challenge.