# Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings

arXiv: https://arxiv.org/abs/2305.09316 · HTML: https://mlsanity.com/paper/2305.09316/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 87 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks](https://mlsanity.com/paper/1909.01716/) (2019)
- [Simple Unsupervised Keyphrase Extraction using Sentence Embeddings](https://mlsanity.com/paper/1801.04470/) (2018)
- [Learning Rich Representation of Keyphrases from Text](https://mlsanity.com/paper/2112.08547/) (2021)
- [Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching](https://mlsanity.com/paper/2004.12297/) (2020)
- [Efficient Classification of Long Documents Using Transformers](https://mlsanity.com/paper/2203.11258/) (2022)
- [Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks](https://mlsanity.com/paper/2007.11192/) (2020)
- [DivGraphPointer: A Graph Pointer Network for Extracting Diverse Keyphrases](https://mlsanity.com/paper/1905.07689/) (2019)
- [TNT-KID: Transformer-based Neural Tagger for Keyword Identification](https://mlsanity.com/paper/2003.09166/) (2020)
- [Heterogeneous Graph Neural Networks for Keyphrase Generation](https://mlsanity.com/paper/2109.04703/) (2021)
- [LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents](https://mlsanity.com/paper/2203.15349/) (2022)
- [Keyphrase Generation Beyond the Boundaries of Title and Abstract](https://mlsanity.com/paper/2112.06776/) (2021)
- [S2ORC: The Semantic Scholar Open Research Corpus](https://mlsanity.com/paper/1911.02782/) (2019)
- [Keyphrase Generation: A Multi-Aspect Survey](https://mlsanity.com/paper/1910.05059/) (2019)

## Built on by (most cited first, 3 of 3)

- [LongKey: Keyphrase Extraction for Long Documents](https://mlsanity.com/paper/2411.17863/) (2024)
- [Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings](https://mlsanity.com/paper/2606.10716/) (2026)
- [Semantic Networks as Clues: A Theoretical Foundation and Process Optimization for Semantic Network Construction](https://mlsanity.com/paper/2608.01936/) (2026)

## Abstract (arXiv)

In this study, we investigate using graph neural network (GNN) representations to enhance contextualized representations of pre-trained language models (PLMs) for keyphrase extraction from lengthy documents. We show that augmenting a PLM with graph embeddings provides a more comprehensive semantic understanding of words in a document, particularly for long documents. We construct a co-occurrence graph of the text and embed it using a graph convolutional network (GCN) trained on the task of edge prediction. We propose a graph-enhanced sequence tagging architecture that augments contextualized PLM embeddings with graph representations. Evaluating on benchmark datasets, we demonstrate that enhancing PLMs with graph embeddings outperforms state-of-the-art models on long documents, showing significant improvements in F1 scores across all the datasets. Our study highlights the potential of GNN representations as a complementary approach to improve PLM performance for keyphrase extraction from long documents.