# Task-specific Pre-training and Prompt Decomposition for Knowledge Graph Population with Language Models

arXiv: https://arxiv.org/abs/2208.12539 · HTML: https://mlsanity.com/paper/2208.12539/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Prompting as Probing: Using Language Models for Knowledge Base Construction](https://mlsanity.com/paper/2208.11057/) (2022)

## Built on by (most cited first, 6 of 6)

- [DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers](https://mlsanity.com/paper/2402.16914/) (2024)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Models](https://mlsanity.com/paper/2308.13207/) (2023)
- [Textual Entailment for Effective Triple Validation in Object Prediction](https://mlsanity.com/paper/2401.16293/) (2024)
- [UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing](https://mlsanity.com/paper/2404.01253/) (2024)
- [Expanding the Vocabulary of BERT for Knowledge Base Construction](https://mlsanity.com/paper/2310.08291/) (2023)

## Abstract (arXiv)

We present a system for knowledge graph population with Language Models, evaluated on the Knowledge Base Construction from Pre-trained Language Models (LM-KBC) challenge at ISWC 2022. Our system involves task-specific pre-training to improve LM representation of the masked object tokens, prompt decomposition for progressive generation of candidate objects, among other methods for higher-quality retrieval. Our system is the winner of track 1 of the LM-KBC challenge, based on BERT LM; it achieves 55.0% F-1 score on the hidden test set of the challenge.