# Phraseformer: Multimodal Key-phrase Extraction using Transformer and Graph Embedding

arXiv: https://arxiv.org/abs/2106.04939 · HTML: https://mlsanity.com/paper/2106.04939/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Unsupervised Keyphrase Extraction with Multipartite Graphs](https://mlsanity.com/paper/1803.08721/) (2018)
- [A Review of Keyphrase Extraction](https://mlsanity.com/paper/1905.05044/) (2019)
- [TNT-KID: Transformer-based Neural Tagger for Keyword Identification](https://mlsanity.com/paper/2003.09166/) (2020)
- [Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings](https://mlsanity.com/paper/1910.08840/) (2019)
- [A Survey on Recent Advances in Sequence Labeling from Deep Learning Models](https://mlsanity.com/paper/2011.06727/) (2020)
- [Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT](https://mlsanity.com/paper/1910.05786/) (2019)
- [Theme-weighted Ranking of Keywords from Text Documents using Phrase Embeddings](https://mlsanity.com/paper/1807.05962/) (2018)
- [BERTERS: Multimodal Representation Learning for Expert Recommendation System with Transformer](https://mlsanity.com/paper/2007.07229/) (2020)
- [TopicBERT: A Transformer transfer learning based memory-graph approach for multimodal streaming social media topic detection](https://mlsanity.com/paper/2008.06877/) (2020)
- [Sequence Labeling: A Practical Approach](https://mlsanity.com/paper/1808.03926/) (2018)
- [ExEm: Expert Embedding using dominating set theory with deep learning approaches](https://mlsanity.com/paper/2001.08503/) (2020)

## Built on by (most cited first, 4 of 4)

- [MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2110.06651/) (2021)
- [From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey](https://mlsanity.com/paper/2305.02579/) (2023)
- [Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings](https://mlsanity.com/paper/2305.09316/) (2023)
- [Each Attribute Matters: Contrastive Attention for Sentence-based Image Editing](https://mlsanity.com/paper/2110.11159/) (2021)

## Abstract (arXiv)

Background: Keyword extraction is a popular research topic in the field of natural language processing. Keywords are terms that describe the most relevant information in a document. The main problem that researchers are facing is how to efficiently and accurately extract the core keywords from a document. However, previous keyword extraction approaches have utilized the text and graph features, there is the lack of models that can properly learn and combine these features in a best way. Methods: In this paper, we develop a multimodal Key-phrase extraction approach, namely Phraseformer, using transformer and graph embedding techniques. In Phraseformer, each keyword candidate is presented by a vector which is the concatenation of the text and structure learning representations. Phraseformer takes the advantages of recent researches such as BERT and ExEm to preserve both representations. Also, the Phraseformer treats the key-phrase extraction task as a sequence labeling problem solved using classification task. Results: We analyze the performance of Phraseformer on three datasets including Inspec, SemEval2010 and SemEval 2017 by F1-score. Also, we investigate the performance of different classifiers on Phraseformer method over Inspec dataset. Experimental results demonstrate the effectiveness of Phraseformer method over the three datasets used. Additionally, the Random Forest classifier gain the highest F1-score among all classifiers. Conclusions: Due to the fact that the combination of BERT and ExEm is more meaningful and can better represent the semantic of words. Hence, Phraseformer significantly outperforms single-modality methods.