# Local Self-Attention over Long Text for Efficient Document Retrieval

arXiv: https://arxiv.org/abs/2005.04908 · HTML: https://mlsanity.com/paper/2005.04908/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 18 of 18)

- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankers](https://mlsanity.com/paper/2104.13640/) (2021)
- [Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges](https://mlsanity.com/paper/2403.00784/) (2024)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models](https://mlsanity.com/paper/2106.05505/) (2021)
- [Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking](https://mlsanity.com/paper/2105.09816/) (2021)
- [SeDR: Segment Representation Learning for Long Documents Dense Retrieval](https://mlsanity.com/paper/2211.10841/) (2022)
- [LADER: Log-Augmented DEnse Retrieval for Biomedical Literature Search](https://mlsanity.com/paper/2304.04590/) (2023)
- [Improving Transformer-Kernel Ranking Model Using Conformer and Query Term Independence](https://mlsanity.com/paper/2104.09393/) (2021)
- [Query-driven Segment Selection for Ranking Long Documents](https://mlsanity.com/paper/2109.04611/) (2021)
- [Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?](https://mlsanity.com/paper/2201.08721/) (2022)
- [COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Tasks](https://mlsanity.com/paper/2204.09593/) (2022)
- [Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systems](https://mlsanity.com/paper/2206.12993/) (2022)
- [QOG:Question and Options Generation based on Language Model](https://mlsanity.com/paper/2406.12381/) (2024)
- [QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking](https://mlsanity.com/paper/2510.11589/) (2025)

## Abstract (arXiv)

Neural networks, particularly Transformer-based architectures, have achieved significant performance improvements on several retrieval benchmarks. When the items being retrieved are documents, the time and memory cost of employing Transformers over a full sequence of document terms can be prohibitive. A popular strategy involves considering only the first n terms of the document. This can, however, result in a biased system that under retrieves longer documents. In this work, we propose a local self-attention which considers a moving window over the document terms and for each term attends only to other terms in the same window. This local attention incurs a fraction of the compute and memory cost of attention over the whole document. The windowed approach also leads to more compact packing of padded documents in minibatches resulting in additional savings. We also employ a learned saturation function and a two-staged pooling strategy to identify relevant regions of the document. The Transformer-Kernel pooling model with these changes can efficiently elicit relevance information from documents with thousands of tokens. We benchmark our proposed modifications on the document ranking task from the TREC 2019 Deep Learning track and observe significant improvements in retrieval quality as well as increased retrieval of longer documents at moderate increase in compute and memory costs.