mlsanityData to 7 Oct 2026

LEGAL-BERT: The Muppets straight out of Law School

arXiv 2010.02559 ↗First posted 6 Oct 2020cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 21; 2023: 30; 2024: 17; 2025: 16; 2026: 6.

111 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  4. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  5. 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
  6. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 61
  7. 2022Legal Prompting: Teaching a Language Model to Think Like a Lawyercited by 60
  8. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  9. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  10. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  11. 2022Mining Legal Arguments in Court Decisionscited by 35
  12. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33
38 more
  1. 2023GeoGalactica: A Scientific Large Language Model in Geosciencecited by 32
  2. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  3. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  4. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  5. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  6. 2021Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Frameworkcited by 21
  7. 2023A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontierscited by 21
  8. 2023Stance Detection: A Practical Guide to Classifying Political Beliefs in Textcited by 20
  9. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  10. 2023Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreementcited by 17
  11. 2023Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoningcited by 17
  12. 2022On the Role of Negative Precedent in Legal Outcome Predictioncited by 16
  13. 2023Lightweight Transformers for Clinical Natural Language Processingcited by 15
  14. 2021AutoLAW: Augmented Legal Reasoning through Legal Precedent Predictioncited by 14
  15. 2023Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Casescited by 14
  16. 2023How well do SOTA legal reasoning models support abductive reasoning?cited by 13
  17. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  18. 2022SsciBERT: A Pre-trained Language Model for Social Science Textscited by 12
  19. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 11
  20. 2025Automatic Legal Writing Evaluation of LLMscited by 11
  21. 2023Zero-Shot Learning for Requirements Classification: An Exploratory Studycited by 9
  22. 2023LexGPT 0.1: pre-trained GPT-J models with Pile of Lawcited by 9
  23. 2024Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settingscited by 9
  24. 2022Fine-grained Intent Classification in the Legal Domaincited by 8
  25. 2022The Legal Argument Reasoning Task in Civil Procedurecited by 8
  26. 2025Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorizationcited by 8
  27. 2021Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviewscited by 7
  28. 2021Linguistically Informed Masking for Representation Learning in the Patent Domaincited by 7
  29. 2022AraLegal-BERT: A pretrained language model for Arabic Legal textcited by 7
  30. 2022LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehensioncited by 7
  31. 2023SemEval 2023 Task 6: LegalEval - Understanding Legal Textscited by 7
  32. 2023Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarizationcited by 7
  33. 2021PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptorscited by 6
  34. 2021Legal Transformer Models May Not Always Helpcited by 6
  35. 2021Unsupervised Law Article Mining based on Deep Pre-Trained Language Representation Models with Application to the Italian Civil Codecited by 6
  36. 2022Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gitacited by 6
  37. 2023Learning Representations on Logs for AIOpscited by 6
  38. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6

50 most cited of 111 All 111 (JSON)