# Supervised Topical Key Phrase Extraction of News Stories using Crowdsourcing, Light Filtering and Co-reference Normalization

arXiv: https://arxiv.org/abs/1306.4886 · HTML: https://mlsanity.com/paper/1306.4886/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2013.

## Built on by (most cited first, 10 of 10)

- [From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey](https://mlsanity.com/paper/2305.02579/) (2023)
- [Keyphrase Generation Beyond the Boundaries of Title and Abstract](https://mlsanity.com/paper/2112.06776/) (2021)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [LLMs Perform Poorly at Concept Extraction in Cyber-security Research Literature](https://mlsanity.com/paper/2312.07110/) (2023)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [Multi-Document Keyphrase Extraction: Dataset, Baselines and Review](https://mlsanity.com/paper/2110.01073/) (2021)
- [Using virtual edges to extract keywords from texts modeled as complex networks](https://mlsanity.com/paper/2205.02172/) (2022)
- [Improving Performance of Automatic Keyword Extraction (AKE) Methods Using PoS-Tagging and Enhanced Semantic-Awareness](https://mlsanity.com/paper/2211.05031/) (2022)
- [Cross-Domain Keyword Extraction with Keyness Patterns](https://mlsanity.com/paper/2409.18724/) (2024)
- [Semantic Networks as Clues: A Theoretical Foundation and Process Optimization for Semantic Network Construction](https://mlsanity.com/paper/2608.01936/) (2026)

## Abstract (arXiv)

Fast and effective automated indexing is critical for search and personalized services. Key phrases that consist of one or more words and represent the main concepts of the document are often used for the purpose of indexing. In this paper, we investigate the use of additional semantic features and pre-processing steps to improve automatic key phrase extraction. These features include the use of signal words and freebase categories. Some of these features lead to significant improvements in the accuracy of the results. We also experimented with 2 forms of document pre-processing that we call light filtering and co-reference normalization. Light filtering removes sentences from the document, which are judged peripheral to its main content. Co-reference normalization unifies several written forms of the same named entity into a unique form. We also needed a "Gold Standard" - a set of labeled documents for training and evaluation. While the subjective nature of key phrase selection precludes a true "Gold Standard", we used Amazon's Mechanical Turk service to obtain a useful approximation. Our data indicates that the biggest improvements in performance were due to shallow semantic features, news categories, and rhetorical signals (nDCG 78.47% vs. 68.93%). The inclusion of deeper semantic features such as Freebase sub-categories was not beneficial by itself, but in combination with pre-processing, did cause slight improvements in the nDCG scores.