# Large Language Models are legal but they are not: Making the case for a powerful LegalLLM arXiv: https://arxiv.org/abs/2311.08890 ยท HTML: https://mlsanity.com/paper/2311.08890/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 37 references that are in the map. - First posted on arXiv 15 Nov 2023. ## Builds on (most cited first, 28 of 28) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019) - [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020) - [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023) - [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021) - [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021) - [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022) - [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021) - [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023) - [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023) - [A Short Survey of Viewing Large Language Models in Legal Aspect](https://mlsanity.com/paper/2303.09136/) (2023) - [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022) - [ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark](https://mlsanity.com/paper/2304.12202/) (2023) - [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023) - [ERNIE-Doc: A Retrospective Long-Document Modeling Transformer](https://mlsanity.com/paper/2012.15688/) (2020) - [Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer](https://mlsanity.com/paper/2211.00974/) (2022) - [PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models](https://mlsanity.com/paper/2309.10238/) (2023) - [LexGPT 0.1: pre-trained GPT-J models with Pile of Law](https://mlsanity.com/paper/2306.05431/) (2023) ## Built on by (most cited first, 12 of 12) - [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025) - [Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topology](https://mlsanity.com/paper/2502.17026/) (2025) - [FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.15800/) (2025) - [Neurosymbolic AI approach to Attribution in Large Language Models](https://mlsanity.com/paper/2410.03726/) (2024) - [Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks](https://mlsanity.com/paper/2407.03895/) (2024) - [Economy Watchers Survey Provides Datasets and Tasks for Japanese Financial Domain](https://mlsanity.com/paper/2407.14727/) (2024) - [Refined and Segmented Price Sentiment Indices from Survey Comments](https://mlsanity.com/paper/2411.09937/) (2024) - [Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice](https://mlsanity.com/paper/2504.12545/) (2025) - [Automatic Information Extraction From Employment Tribunal Judgements Using Large Language Models](https://mlsanity.com/paper/2403.12936/) (2024) - [An Integrated Framework of Prompt Engineering and Multidimensional Knowledge Graphs for Legal Dispute Analysis](https://mlsanity.com/paper/2507.07893/) (2025) - [The Expert Validation Framework (EVF): Enabling Domain Expert Control in AI Engineering](https://mlsanity.com/paper/2601.12327/) (2026) - [Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining](https://mlsanity.com/paper/2609.26034/) (2026) ## Abstract (arXiv) Realizing the recent advances in Natural Language Processing (NLP) to the legal sector poses challenging problems such as extremely long sequence lengths, specialized vocabulary that is usually only understood by legal professionals, and high amounts of data imbalance. The recent surge of Large Language Models (LLMs) has begun to provide new opportunities to apply NLP in the legal domain due to their ability to handle lengthy, complex sequences. Moreover, the emergence of domain-specific LLMs has displayed extremely promising results on various tasks. In this study, we aim to quantify how general LLMs perform in comparison to legal-domain models (be it an LLM or otherwise). Specifically, we compare the zero-shot performance of three general-purpose LLMs (ChatGPT-20b, LLaMA-2-70b, and Falcon-180b) on the LEDGAR subset of the LexGLUE benchmark for contract provision classification. Although the LLMs were not explicitly trained on legal data, we observe that they are still able to classify the theme correctly in most cases. However, we find that their mic-F1/mac-F1 performance is up to 19.2/26.8% lesser than smaller models fine-tuned on the legal domain, thus underscoring the need for more powerful legal-domain LLMs.