# Keyphrase Prediction With Pre-trained Language Model

arXiv: https://arxiv.org/abs/2004.10462 · HTML: https://mlsanity.com/paper/2004.10462/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [Learning Rich Representation of Keyphrases from Text](https://mlsanity.com/paper/2112.08547/) (2021)
- [MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2110.06651/) (2021)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [Hyperbolic Relevance Matching for Neural Keyphrase Extraction](https://mlsanity.com/paper/2205.02047/) (2022)
- [Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation](https://mlsanity.com/paper/2211.01981/) (2022)
- [KPDrop: Improving Absent Keyphrase Generation](https://mlsanity.com/paper/2112.01476/) (2021)
- [SimCKP: Simple Contrastive Learning of Keyphrase Representations](https://mlsanity.com/paper/2310.08221/) (2023)
- [On Leveraging Encoder-only Pre-trained Language Models for Effective Keyphrase Generation](https://mlsanity.com/paper/2402.14052/) (2024)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [Deep Keyphrase Completion](https://mlsanity.com/paper/2111.01910/) (2021)
- [Abstractive Opinion Tagging](https://mlsanity.com/paper/2101.06880/) (2021)
- [Neural Keyphrase Generation: Analysis and Evaluation](https://mlsanity.com/paper/2304.13883/) (2023)
- [Zero-Shot Keyphrase Generation: Investigating Specialized Instructions and Multi-Sample Aggregation on Large Language Models](https://mlsanity.com/paper/2503.00597/) (2025)

## Abstract (arXiv)

Recently, generative methods have been widely used in keyphrase prediction, thanks to their capability to produce both present keyphrases that appear in the source text and absent keyphrases that do not match any source text. However, the absent keyphrases are generated at the cost of the performance on present keyphrase prediction, since previous works mainly use generative models that rely on the copying mechanism and select words step by step. Besides, the extractive model that directly extracts a text span is more suitable for predicting the present keyphrase. Considering the different characteristics of extractive and generative methods, we propose to divide the keyphrase prediction into two subtasks, i.e., present keyphrase extraction (PKE) and absent keyphrase generation (AKG), to fully exploit their respective advantages. On this basis, a joint inference framework is proposed to make the most of BERT in two subtasks. For PKE, we tackle this task as a sequence labeling problem with the pre-trained language model BERT. For AKG, we introduce a Transformer-based architecture, which fully integrates the present keyphrase knowledge learned from PKE by the fine-tuned BERT. The experimental results show that our approach can achieve state-of-the-art results on both tasks on benchmark datasets.