# Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings

arXiv: https://arxiv.org/abs/1910.08840 · HTML: https://mlsanity.com/paper/1910.08840/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2019.

## Built on by (most cited first, 14 of 14)

- [Learning Rich Representation of Keyphrases from Text](https://mlsanity.com/paper/2112.08547/) (2021)
- [MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2110.06651/) (2021)
- [Representation Learning for Resource-Constrained Keyphrase Generation](https://mlsanity.com/paper/2203.08118/) (2022)
- [From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey](https://mlsanity.com/paper/2305.02579/) (2023)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [Out of Thin Air: Is Zero-Shot Cross-Lingual Keyword Detection Better Than Unsupervised?](https://mlsanity.com/paper/2202.06650/) (2022)
- [Phraseformer: Multimodal Key-phrase Extraction using Transformer and Graph Embedding](https://mlsanity.com/paper/2106.04939/) (2021)
- [Extending Neural Keyword Extraction with TF-IDF tagset matching](https://mlsanity.com/paper/2102.00472/) (2021)
- [LLMs Perform Poorly at Concept Extraction in Cyber-security Research Literature](https://mlsanity.com/paper/2312.07110/) (2023)
- [On Leveraging Encoder-only Pre-trained Language Models for Effective Keyphrase Generation](https://mlsanity.com/paper/2402.14052/) (2024)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [Cisco at AAAI-CAD21 shared task: Predicting Emphasis in Presentation Slides using Contextualized Embeddings](https://mlsanity.com/paper/2101.11422/) (2021)
- [SEKE: Specialised Experts for Keyword Extraction](https://mlsanity.com/paper/2412.14087/) (2024)
- [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](https://mlsanity.com/paper/2606.31058/) (2026)

## Abstract (arXiv)

In this paper, we formulate keyphrase extraction from scholarly articles as a sequence labeling task solved using a BiLSTM-CRF, where the words in the input text are represented using deep contextualized embeddings. We evaluate the proposed architecture using both contextualized and fixed word embedding models on three different benchmark datasets (Inspec, SemEval 2010, SemEval 2017) and compare with existing popular unsupervised and supervised techniques. Our results quantify the benefits of (a) using contextualized embeddings (e.g. BERT) over fixed word embeddings (e.g. Glove); (b) using a BiLSTM-CRF architecture with contextualized word embeddings over fine-tuning the contextualized word embedding model directly, and (c) using genre-specific contextualized embeddings (SciBERT). Through error analysis, we also provide some insights into why particular models work better than others. Lastly, we present a case study where we analyze different self-attention layers of the two best models (BERT and SciBERT) to better understand the predictions made by each for the task of keyphrase extraction.