# Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer

arXiv: https://arxiv.org/abs/2211.00974 · HTML: https://mlsanity.com/paper/2211.00974/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service](https://mlsanity.com/paper/1805.01217/) (2018)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching](https://mlsanity.com/paper/2004.12297/) (2020)
- [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022)
- [ERNIE-Doc: A Retrospective Long-Document Modeling Transformer](https://mlsanity.com/paper/2012.15688/) (2020)
- [Corpus for Automatic Structuring of Legal Documents](https://mlsanity.com/paper/2201.13125/) (2022)
- [Obligation and Prohibition Extraction Using Hierarchical RNNs](https://mlsanity.com/paper/1805.03871/) (2018)
- [An Empirical Study on Cross-X Transfer for Legal Judgment Prediction](https://mlsanity.com/paper/2209.12325/) (2022)
- [How not to Lie with a Benchmark: Rearranging NLP Leaderboards](https://mlsanity.com/paper/2112.01342/) (2021)
- [Using Centroids of Word Embeddings and Word Mover's Distance for Biomedical Document Retrieval in Question Answering](https://mlsanity.com/paper/1608.03905/) (2016)

## Built on by (most cited first, 20 of 20)

- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [Large Language Models are legal but they are not: Making the case for a powerful LegalLLM](https://mlsanity.com/paper/2311.08890/) (2023)
- [LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2402.04335/) (2024)
- [Legal Extractive Summarization of U.S. Court Opinions](https://mlsanity.com/paper/2305.08428/) (2023)
- [LexGPT 0.1: pre-trained GPT-J models with Pile of Law](https://mlsanity.com/paper/2306.05431/) (2023)
- [Large Language Model Prompt Chaining for Long Legal Document Classification](https://mlsanity.com/paper/2308.04138/) (2023)
- [LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2410.12064/) (2024)
- [Summarizing long regulatory documents with a multi-step pipeline](https://mlsanity.com/paper/2408.09777/) (2024)
- [Unfair TOS: An Automated Approach using Customized BERT](https://mlsanity.com/paper/2401.11207/) (2024)
- [Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges](https://mlsanity.com/paper/2404.02269/) (2024)
- [The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines](https://mlsanity.com/paper/2401.11852/) (2024)
- [Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification](https://mlsanity.com/paper/2605.17691/) (2026)
- [Legal NLP Meets MiCAR: Advancing the Analysis of Crypto White Papers](https://mlsanity.com/paper/2310.10333/) (2023)
- [A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval](https://mlsanity.com/paper/2507.01058/) (2025)
- [Skim-Aware Contrastive Learning for Efficient Document Representation](https://mlsanity.com/paper/2512.24373/) (2025)
- [Classifying long legal documents using short random chunks](https://mlsanity.com/paper/2512.24997/) (2025)
- [Legal Experts Disagree With Rationale Extraction Techniques for Explaining ECtHR Case Outcome Classification](https://mlsanity.com/paper/2601.12419/) (2026)
- [A Two-Stage Architecture for NDA Analysis: LLM-based Segmentation and Transformer-based Clause Classification](https://mlsanity.com/paper/2603.09990/) (2026)
- [LLM-Assisted Causal Structure Disambiguation and Factor Extraction for Legal Judgment Prediction](https://mlsanity.com/paper/2603.11446/) (2026)

## Abstract (arXiv)

Pre-trained Transformers currently dominate most NLP tasks. They impose, however, limits on the maximum input length (512 sub-words in BERT), which are too restrictive in the legal domain. Even sparse-attention models, such as Longformer and BigBird, which increase the maximum input length to 4,096 sub-words, severely truncate texts in three of the six datasets of LexGLUE. Simpler linear classifiers with TF-IDF features can handle texts of any length, require far less resources to train and deploy, but are usually outperformed by pre-trained Transformers. We explore two directions to cope with long legal texts: (i) modifying a Longformer warm-started from LegalBERT to handle even longer texts (up to 8,192 sub-words), and (ii) modifying LegalBERT to use TF-IDF representations. The first approach is the best in terms of performance, surpassing a hierarchical version of LegalBERT, which was the previous state of the art in LexGLUE. The second approach leads to computationally more efficient models at the expense of lower performance, but the resulting models still outperform overall a linear SVM with TF-IDF features in long legal document classification.