# Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

arXiv: https://arxiv.org/abs/2004.12297 · HTML: https://mlsanity.com/paper/2004.12297/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2020.

## Built on by (most cited first, 30 of 35)

- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain](https://mlsanity.com/paper/2210.13448/) (2022)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling](https://mlsanity.com/paper/2106.01040/) (2021)
- [WebFormer: The Web-page Transformer for Structure Information Extraction](https://mlsanity.com/paper/2202.00217/) (2022)
- [VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups](https://mlsanity.com/paper/2106.00676/) (2021)
- [Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer](https://mlsanity.com/paper/2211.00974/) (2022)
- [CDLM: Cross-Document Language Modeling](https://mlsanity.com/paper/2101.00406/) (2021)
- [RoR: Read-over-Read for Long Document Machine Reading Comprehension](https://mlsanity.com/paper/2109.04780/) (2021)
- [Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey](https://mlsanity.com/paper/2201.09267/) (2022)
- [Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networks](https://mlsanity.com/paper/2301.12847/) (2023)
- [LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction](https://mlsanity.com/paper/2305.19585/) (2023)
- [Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification](https://mlsanity.com/paper/2406.01283/) (2024)
- [Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification](https://mlsanity.com/paper/2104.08444/) (2021)
- [Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT](https://mlsanity.com/paper/2310.20558/) (2023)
- [Multi-view Content-aware Indexing for Long Document Retrieval](https://mlsanity.com/paper/2404.15103/) (2024)
- [Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings](https://mlsanity.com/paper/2305.09316/) (2023)
- [Can Deep Neural Networks Predict Data Correlations from Column Names?](https://mlsanity.com/paper/2107.04553/) (2021)
- [Machine Learning for Violence Risk Assessment Using Dutch Clinical Notes](https://mlsanity.com/paper/2204.13535/) (2022)
- [Attention over pre-trained Sentence Embeddings for Long Document Classification](https://mlsanity.com/paper/2307.09084/) (2023)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)
- [Generalization in Healthcare AI: Evaluation of a Clinical Large Language Model](https://mlsanity.com/paper/2402.10965/) (2024)
- [SAE-FiRE: Enhancing Earnings Surprise Predictions Through Sparse Autoencoder Feature Selection](https://mlsanity.com/paper/2505.14420/) (2025)
- [FETILDA: An Effective Framework For Fin-tuned Embeddings For Long Financial Text Documents](https://mlsanity.com/paper/2206.06952/) (2022)
- [Legal Element-oriented Modeling with Multi-view Contrastive Learning for Legal Case Retrieval](https://mlsanity.com/paper/2210.05188/) (2022)
- [HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2310.09720/) (2023)
- [Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning](https://mlsanity.com/paper/2311.17514/) (2023)

## Abstract (arXiv)

Many natural language processing and information retrieval problems can be formalized as the task of semantic matching. Existing work in this area has been largely focused on matching between short texts (e.g., question answering), or between a short and a long text (e.g., ad-hoc retrieval). Semantic matching between long-form documents, which has many important applications like news recommendation, related article recommendation and document clustering, is relatively less explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have achieved state-of-the-art performance in the task of text matching. These models, however, are still limited to short text like a few sentences or one paragraph due to the quadratic computational complexity of self-attention with respect to input text length. In this paper, we address the issue by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for long-form document matching. Our model contains several innovations to adapt self-attention models for longer text input. In order to better capture sentence level semantic relations within a document, we pre-train the model with a novel masked sentence block language modeling task in addition to the masked word language modeling task used by BERT. Our experimental results on several benchmark datasets for long-form document matching show that our proposed SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT. Comparing to BERT based baselines, our model is able to increase maximum input text length from 512 to 2048. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching.