# Language Model as an Annotator: Unsupervised Context-aware Quality Phrase Generation

arXiv: https://arxiv.org/abs/2312.17349 · HTML: https://mlsanity.com/paper/2312.17349/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://mlsanity.com/paper/2210.09932/) (2022)
- [KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documents](https://mlsanity.com/paper/1911.12559/) (2019)
- [HiExpan: Task-Guided Taxonomy Construction by Hierarchical Tree Expansion](https://mlsanity.com/paper/1910.08194/) (2019)
- [UCPhrase: Unsupervised Context-aware Quality Phrase Tagging](https://mlsanity.com/paper/2105.14078/) (2021)
- [Applying a Generic Sequence-to-Sequence Model for Simple and Effective Keyphrase Generation](https://mlsanity.com/paper/2201.05302/) (2022)
- [Scalable Topical Phrase Mining from Text Corpora](https://mlsanity.com/paper/1406.6312/) (2014)
- [Large-Scale Evaluation of Keyphrase Extraction Models](https://mlsanity.com/paper/2003.04628/) (2020)
- [PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation](https://mlsanity.com/paper/2210.03496/) (2022)

## Built on by (most cited first, 1 of 1)

- [Neurosymbolic Graph Enrichment for Grounded World Models](https://mlsanity.com/paper/2411.12671/) (2024)

## Abstract (arXiv)

Phrase mining is a fundamental text mining task that aims to identify quality phrases from context. Nevertheless, the scarcity of extensive gold labels datasets, demanding substantial annotation efforts from experts, renders this task exceptionally challenging. Furthermore, the emerging, infrequent, and domain-specific nature of quality phrases presents further challenges in dealing with this task. In this paper, we propose LMPhrase, a novel unsupervised context-aware quality phrase mining framework built upon large pre-trained language models (LMs). Specifically, we first mine quality phrases as silver labels by employing a parameter-free probing technique called Perturbed Masking on the pre-trained language model BERT (coined as Annotator). In contrast to typical statistic-based or distantly-supervised methods, our silver labels, derived from large pre-trained language models, take into account rich contextual information contained in the LMs. As a result, they bring distinct advantages in preserving informativeness, concordance, and completeness of quality phrases. Secondly, training a discriminative span prediction model heavily relies on massive annotated data and is likely to face the risk of overfitting silver labels. Alternatively, we formalize phrase tagging task as the sequence generation problem by directly fine-tuning on the Sequence-to-Sequence pre-trained language model BART with silver labels (coined as Generator). Finally, we merge the quality phrases from both the Annotator and Generator as the final predictions, considering their complementary nature and distinct characteristics. Extensive experiments show that our LMPhrase consistently outperforms all the existing competitors across two different granularity phrase mining tasks, where each task is tested on two different domain datasets.