# AdaptKeyBERT: An Attention-Based approach towards Few-Shot & Zero-Shot Domain Adaptation of KeyBERT

arXiv: https://arxiv.org/abs/2211.07499 · HTML: https://mlsanity.com/paper/2211.07499/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 18 references that are in the map.
- First posted on arXiv 14 Nov 2022.

## Builds on (most cited first, 3 of 3)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Zero-shot Relation Classification from Side Information](https://mlsanity.com/paper/2011.07126/) (2020)
- [Out of Thin Air: Is Zero-Shot Cross-Lingual Keyword Detection Better Than Unsupervised?](https://mlsanity.com/paper/2202.06650/) (2022)

## Built on by (most cited first, 2 of 2)

- [LLM-TAKE: Theme Aware Keyword Extraction Using Large Language Models](https://mlsanity.com/paper/2312.00909/) (2023)
- [Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2310.16959/) (2023)

## Abstract (arXiv)

Keyword extraction has been an important topic for modern natural language processing. With its applications ranging from ontology generation, fact verification in summarized text, and recommendation systems. While it has had significant data-intensive applications, it is often hampered when the data set is small. Downstream training for keyword extractors is a lengthy process and requires a significant amount of data. Recently, Few-shot Learning (FSL) and Zero-Shot Learning (ZSL) have been proposed to tackle this problem. Therefore, we propose AdaptKeyBERT, a pipeline for training keyword extractors with LLM bases by incorporating the concept of regularized attention into a pre-training phase for downstream domain adaptation. As we believe our work has implications to be utilized in the pipeline of FSL/ZSL and keyword extraction, we open-source our code as well as provide the fine-tuning library of the same name AdaptKeyBERT at https://github.com/AmanPriyanshu/AdaptKeyBERT.