# MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction

arXiv: https://arxiv.org/abs/2110.06651 · HTML: https://mlsanity.com/paper/2110.06651/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [Simple Unsupervised Keyphrase Extraction using Sentence Embeddings](https://mlsanity.com/paper/1801.04470/) (2018)
- [Unsupervised Keyphrase Extraction with Multipartite Graphs](https://mlsanity.com/paper/1803.08721/) (2018)
- [An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation](https://mlsanity.com/paper/1607.05368/) (2016)
- [Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction](https://mlsanity.com/paper/2002.00737/) (2020)
- [A Review of Keyphrase Extraction](https://mlsanity.com/paper/1905.05044/) (2019)
- [An Integrated Approach for Keyphrase Generation via Exploring the Power of Retrieval and Extraction](https://mlsanity.com/paper/1904.03454/) (2019)
- [An Empirical Study on Neural Keyphrase Generation](https://mlsanity.com/paper/2009.10229/) (2020)
- [TNT-KID: Transformer-based Neural Tagger for Keyword Identification](https://mlsanity.com/paper/2003.09166/) (2020)
- [Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings](https://mlsanity.com/paper/1910.08840/) (2019)
- [Keyphrase Prediction With Pre-trained Language Model](https://mlsanity.com/paper/2004.10462/) (2020)

## Built on by (most cited first, 20 of 20)

- [From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey](https://mlsanity.com/paper/2305.02579/) (2023)
- [LeCoPCR: Legal Concept-guided Prior Case Retrieval for European Court of Human Rights cases](https://mlsanity.com/paper/2501.14114/) (2025)
- [SimCKP: Simple Contrastive Learning of Keyphrase Representations](https://mlsanity.com/paper/2310.08221/) (2023)
- [Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Study](https://mlsanity.com/paper/2312.15156/) (2023)
- [Theme-driven Keyphrase Extraction to Analyze Social Media Discourse](https://mlsanity.com/paper/2301.11508/) (2023)
- [MUG: A General Meeting Understanding and Generation Benchmark](https://mlsanity.com/paper/2303.13939/) (2023)
- [A Preliminary Empirical Study on Prompt-based Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2405.16571/) (2024)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [One2set + Large Language Model: Best Partners for Keyphrase Generation](https://mlsanity.com/paper/2410.03421/) (2024)
- [Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction](https://mlsanity.com/paper/2308.08739/) (2023)
- [Keyword Extraction, and Aspect Classification in Sinhala, English, and Code-Mixed Content](https://mlsanity.com/paper/2504.10679/) (2025)
- [Unsupervised Keyphrase Extraction via Interpretable Neural Networks](https://mlsanity.com/paper/2203.07640/) (2022)
- [MAPEX: A Multi-Agent Pipeline for Keyphrase Extraction](https://mlsanity.com/paper/2509.18813/) (2025)
- [Improving Performance of Automatic Keyword Extraction (AKE) Methods Using PoS-Tagging and Enhanced Semantic-Awareness](https://mlsanity.com/paper/2211.05031/) (2022)
- [Topic-to-essay generation with knowledge-based content selection](https://mlsanity.com/paper/2402.16248/) (2024)
- [Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2409.10907/) (2024)
- [ACE: A Cardinality Estimator for Set-Valued Queries](https://mlsanity.com/paper/2503.14929/) (2025)
- [ERU-KG: Efficient Reference-aligned Unsupervised Keyphrase Generation](https://mlsanity.com/paper/2505.24219/) (2025)
- [Enhancing Unsupervised Keyword Extraction in Academic Papers through Integrating Highlights with Abstract](https://mlsanity.com/paper/2604.19505/) (2026)
- [Semantic Networks as Clues: A Theoretical Foundation and Process Optimization for Semantic Network Construction](https://mlsanity.com/paper/2608.01936/) (2026)

## Abstract (arXiv)

Keyphrase extraction (KPE) automatically extracts phrases in a document that provide a concise summary of the core content, which benefits downstream information retrieval and NLP tasks. Previous state-of-the-art (SOTA) methods select candidate keyphrases based on the similarity between learned representations of the candidates and the document. They suffer performance degradation on long documents due to discrepancy between sequence lengths which causes mismatch between representations of keyphrase candidates and the document. In this work, we propose a novel unsupervised embedding-based KPE approach, Masked Document Embedding Rank (MDERank), to address this problem by leveraging a mask strategy and ranking candidates by the similarity between embeddings of the source document and the masked document. We further develop a KPE-oriented BERT (KPEBERT) model by proposing a novel self-supervised contrastive learning method, which is more compatible to MDERank than vanilla BERT. Comprehensive evaluations on six KPE benchmarks demonstrate that the proposed MDERank outperforms state-of-the-art unsupervised KPE approach by average 1.80 F1@15 improvement. MDERank further benefits from KPEBERT and overall achieves average 3.53 F1@15 improvement over the SOTA SIFRank. Our code is available at urlhttps://github.com/LinhanZ/mderank.