# Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference

arXiv: https://arxiv.org/abs/2106.01186 · HTML: https://mlsanity.com/paper/2106.01186/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Neural Attentive Multiview Machines](https://mlsanity.com/paper/2002.07696/) (2020)
- [RecoBERT: A Catalog Language Model for Text-Based Recommendations](https://mlsanity.com/paper/2009.13292/) (2020)
- [Multilevel Text Alignment with Cross-Document Attention](https://mlsanity.com/paper/2010.01263/) (2020)

## Built on by (most cited first, 13 of 13)

- [Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps](https://mlsanity.com/paper/2204.11073/) (2022)
- [Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity](https://mlsanity.com/paper/2111.08366/) (2021)
- [Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering](https://mlsanity.com/paper/2305.15387/) (2023)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Interpreting BERT-based Text Similarity via Activation and Saliency Maps](https://mlsanity.com/paper/2208.06612/) (2022)
- [Cold Item Integration in Deep Hybrid Recommenders via Tunable Stochastic Gates](https://mlsanity.com/paper/2112.07615/) (2021)
- [Paragraph-based Transformer Pre-training for Multi-Sentence Inference](https://mlsanity.com/paper/2205.01228/) (2022)
- [Learning to Explain: A Model-Agnostic Framework for Explaining Black Box Models](https://mlsanity.com/paper/2310.16584/) (2023)
- [MetricBERT: Text Representation Learning via Self-Supervised Triplet Training](https://mlsanity.com/paper/2208.06610/) (2022)
- [Efficient Discovery and Effective Evaluation of Visual Perceptual Similarity: A Benchmark and Beyond](https://mlsanity.com/paper/2308.14753/) (2023)
- [Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning](https://mlsanity.com/paper/2311.17514/) (2023)
- [FastDoc: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy](https://mlsanity.com/paper/2306.06190/) (2023)
- [Skim-Aware Contrastive Learning for Efficient Document Representation](https://mlsanity.com/paper/2512.24373/) (2025)

## Abstract (arXiv)

We present a novel model for the problem of ranking a collection of documents according to their semantic similarity to a source (query) document. While the problem of document-to-document similarity ranking has been studied, most modern methods are limited to relatively short documents or rely on the existence of "ground-truth" similarity labels. Yet, in most common real-world cases, similarity ranking is an unsupervised problem as similarity labels are unavailable. Moreover, an ideal model should not be restricted by documents' length. Hence, we introduce SDR, a self-supervised method for document similarity that can be applied to documents of arbitrary length. Importantly, SDR can be effectively applied to extremely long documents, exceeding the 4,096 maximal token limits of Longformer. Extensive evaluations on large document datasets show that SDR significantly outperforms its alternatives across all metrics. To accelerate future research on unlabeled long document similarity ranking, and as an additional contribution to the community, we herein publish two human-annotated test sets of long documents similarity evaluation. The SDR code and datasets are publicly available.