# Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Study

arXiv: https://arxiv.org/abs/2312.15156 · HTML: https://mlsanity.com/paper/2312.15156/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Simple Unsupervised Keyphrase Extraction using Sentence Embeddings](https://mlsanity.com/paper/1801.04470/) (2018)
- [Unsupervised Keyphrase Extraction with Multipartite Graphs](https://mlsanity.com/paper/1803.08721/) (2018)
- [Open Domain Web Keyphrase Extraction Beyond Language Modeling](https://mlsanity.com/paper/1911.02671/) (2019)
- [MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2110.06651/) (2021)
- [Capturing Global Informativeness in Open Domain Keyphrase Extraction](https://mlsanity.com/paper/2004.13639/) (2020)
- [Unsupervised Keyphrase Extraction by Jointly Modeling Local and Global Context](https://mlsanity.com/paper/2109.07293/) (2021)
- [Is ChatGPT A Good Keyphrase Generator? A Preliminary Study](https://mlsanity.com/paper/2303.13001/) (2023)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Importance Estimation from Multiple Perspectives for Keyphrase Extraction](https://mlsanity.com/paper/2110.09749/) (2021)
- [LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents](https://mlsanity.com/paper/2203.15349/) (2022)
- [Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?](https://mlsanity.com/paper/2311.09175/) (2023)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding](https://mlsanity.com/paper/2210.04105/) (2022)
- [HiPool: Modeling Long Documents Using Graph Neural Networks](https://mlsanity.com/paper/2305.03319/) (2023)

## Built on by (most cited first, 4 of 4)

- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [From Lazy to Prolific: Tackling Missing Labels in Open Vocabulary Extreme Classification by Positive-Unlabeled Sequence Learning](https://mlsanity.com/paper/2408.08981/) (2024)
- [Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases](https://mlsanity.com/paper/2410.18040/) (2024)
- [Zero-Shot Keyphrase Generation: Investigating Specialized Instructions and Multi-Sample Aggregation on Large Language Models](https://mlsanity.com/paper/2503.00597/) (2025)

## Abstract (arXiv)

Zero-shot keyphrase extraction aims to build a keyphrase extractor without training by human-annotated data, which is challenging due to the limited human intervention involved. Challenging but worthwhile, zero-shot setting efficiently reduces the time and effort that data labeling takes. Recent efforts on pre-trained large language models (e.g., ChatGPT and ChatGLM) show promising performance on zero-shot settings, thus inspiring us to explore prompt-based methods. In this paper, we ask whether strong keyphrase extraction models can be constructed by directly prompting the large language model ChatGPT. Through experimental results, it is found that ChatGPT still has a lot of room for improvement in the keyphrase extraction task compared to existing state-of-the-art unsupervised and supervised models.