# Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings

arXiv: https://arxiv.org/abs/2606.10716 · HTML: https://mlsanity.com/paper/2606.10716/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Simple Unsupervised Keyphrase Extraction using Sentence Embeddings](https://mlsanity.com/paper/1801.04470/) (2018)
- [Learning Rich Representation of Keyphrases from Text](https://mlsanity.com/paper/2112.08547/) (2021)
- [PatternRank: Leveraging Pretrained Language Models and Part of Speech for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2210.05245/) (2022)
- [TNT-KID: Transformer-based Neural Tagger for Keyword Identification](https://mlsanity.com/paper/2003.09166/) (2020)
- [LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents](https://mlsanity.com/paper/2203.15349/) (2022)
- [ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task](https://mlsanity.com/paper/2304.14177/) (2023)
- [PromptRank: Unsupervised Keyphrase Extraction Using Prompt](https://mlsanity.com/paper/2305.04490/) (2023)
- [Keyphrase Generation Beyond the Boundaries of Title and Abstract](https://mlsanity.com/paper/2112.06776/) (2021)
- [S2ORC: The Semantic Scholar Open Research Corpus](https://mlsanity.com/paper/1911.02782/) (2019)
- [Theme-weighted Ranking of Keywords from Text Documents using Phrase Embeddings](https://mlsanity.com/paper/1807.05962/) (2018)
- [Query-Based Keyphrase Extraction from Long Documents](https://mlsanity.com/paper/2205.05391/) (2022)
- [Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings](https://mlsanity.com/paper/2305.09316/) (2023)
- [LongKey: Keyphrase Extraction for Long Documents](https://mlsanity.com/paper/2411.17863/) (2024)
- [MAPEX: A Multi-Agent Pipeline for Keyphrase Extraction](https://mlsanity.com/paper/2509.18813/) (2025)

## Abstract (arXiv)

Pre-trained language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations. However, long-document KPE remains challenging because salient keyphrase evidence may be scattered across distant document sections that cannot be jointly captured within the limited context window of most PLMs. Although long-context large language models (LLMs) can process broader textual contexts, their computational cost limits their practicality for efficient and high-throughput KPE. To overcome this limitation, we propose an attention expansion mechanism that augments PLM token representations with information from surrounding out-of-context chunks using pre-trained word embeddings. The proposed mechanism expands the effective contextual scope of PLM-based KPE models without requiring full-document attention or expensive LLM-based inference. We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders, using two training regimes and five benchmark corpora from scientific and news domains. Experimental results demonstrate that attention expansion consistently enhances KPE performance across all evaluation settings, outperforming state-of-the-art models and yielding notable improvements in F1 score. The improvements extend to domain-specific, task-specialized, and native long-context models, showing that the proposed mechanism provides complementary information rather than merely compensating for limited input length. These results establish attention expansion as an efficient and effective strategy for long-document KPE.