# UCPhrase: Unsupervised Context-aware Quality Phrase Tagging

arXiv: https://arxiv.org/abs/2105.14078 · HTML: https://mlsanity.com/paper/2105.14078/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [On the Transformer Growth for Progressive BERT Training](https://mlsanity.com/paper/2010.12562/) (2020)
- [Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction](https://mlsanity.com/paper/2002.00737/) (2020)
- [KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documents](https://mlsanity.com/paper/1911.12559/) (2019)
- [Zipf's law holds for phrases, not words](https://mlsanity.com/paper/1406.5181/) (2014)
- [Scalable Topical Phrase Mining from Text Corpora](https://mlsanity.com/paper/1406.6312/) (2014)
- [Large-Scale Evaluation of Keyphrase Extraction Models](https://mlsanity.com/paper/2003.04628/) (2020)

## Built on by (most cited first, 14 of 14)

- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak Supervision](https://mlsanity.com/paper/2204.13874/) (2022)
- [From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey](https://mlsanity.com/paper/2305.02579/) (2023)
- [AI-native Memory: A Pathway from LLMs Towards AGI](https://mlsanity.com/paper/2406.18312/) (2024)
- [A Cross Attention Approach to Diagnostic Explainability using Clinical Practice Guidelines for Depression](https://mlsanity.com/paper/2311.13852/) (2023)
- [Unsupervised Key Event Detection from Massive Text Corpora](https://mlsanity.com/paper/2206.04153/) (2022)
- [Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation](https://mlsanity.com/paper/2211.01981/) (2022)
- [PDSum: Prototype-driven Continuous Summarization of Evolving Multi-document Sets Stream](https://mlsanity.com/paper/2302.05550/) (2023)
- [Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning](https://mlsanity.com/paper/2308.04712/) (2023)
- [Less than One-shot: Named Entity Recognition via Extremely Weak Supervision](https://mlsanity.com/paper/2311.02861/) (2023)
- [Theme-driven Keyphrase Extraction to Analyze Social Media Discourse](https://mlsanity.com/paper/2301.11508/) (2023)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [Deep Knowledge-Infusion For Explainable Depression Detection](https://mlsanity.com/paper/2409.02122/) (2024)
- [Language Model as an Annotator: Unsupervised Context-aware Quality Phrase Generation](https://mlsanity.com/paper/2312.17349/) (2023)

## Abstract (arXiv)

Identifying and understanding quality phrases from context is a fundamental task in text mining. The most challenging part of this task arguably lies in uncommon, emerging, and domain-specific phrases. The infrequent nature of these phrases significantly hurts the performance of phrase mining methods that rely on sufficient phrase occurrences in the input corpus. Context-aware tagging models, though not restricted by frequency, heavily rely on domain experts for either massive sentence-level gold labels or handcrafted gazetteers. In this work, we propose UCPhrase, a novel unsupervised context-aware quality phrase tagger. Specifically, we induce high-quality phrase spans as silver labels from consistently co-occurring word sequences within each document. Compared with typical context-agnostic distant supervision based on existing knowledge bases (KBs), our silver labels root deeply in the input domain and context, thus having unique advantages in preserving contextual completeness and capturing emerging, out-of-KB phrases. Training a conventional neural tagger based on silver labels usually faces the risk of overfitting phrase surface names. Alternatively, we observe that the contextualized attention maps generated from a transformer-based neural language model effectively reveal the connections between words in a surface-agnostic way. Therefore, we pair such attention maps with the silver labels to train a lightweight span prediction model, which can be applied to new input to recognize (unseen) quality phrases regardless of their surface names or frequency. Thorough experiments on various tasks and datasets, including corpus-level phrase ranking, document-level keyphrase extraction, and sentence-level phrase tagging, demonstrate the superiority of our design over state-of-the-art pre-trained, unsupervised, and distantly supervised methods.