# JuriBERT: A Masked-Language Model Adaptation for French Legal Text

arXiv: https://arxiv.org/abs/2110.01485 · HTML: https://mlsanity.com/paper/2110.01485/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://mlsanity.com/paper/2010.12321/) (2020)
- [Exploring the Use of Text Classification in the Legal Domain](https://mlsanity.com/paper/1710.09306/) (2017)
- [BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding](https://mlsanity.com/paper/1911.00473/) (2019)
- [Evaluation Of Word Embeddings From Large-Scale French Web Content](https://mlsanity.com/paper/2105.01990/) (2021)

## Built on by (most cited first, 14 of 14)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding](https://mlsanity.com/paper/2310.14478/) (2023)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://mlsanity.com/paper/2410.09527/) (2024)
- [ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Framework](https://mlsanity.com/paper/2506.18768/) (2025)
- [Human Centered AI for Indian Legal Text Analytics](https://mlsanity.com/paper/2403.10944/) (2024)
- [The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18645/) (2024)
- [Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions](https://mlsanity.com/paper/2603.22973/) (2026)
- [NLP Research and Resources at DaSciM, Ecole Polytechnique](https://mlsanity.com/paper/2112.00566/) (2021)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)
- [JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences](https://mlsanity.com/paper/2508.16870/) (2025)
- [Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law](https://mlsanity.com/paper/2608.09393/) (2026)

## Abstract (arXiv)

Language models have proven to be very useful when adapted to specific domains. Nonetheless, little research has been done on the adaptation of domain-specific BERT models in the French language. In this paper, we focus on creating a language model adapted to French legal text with the goal of helping law professionals. We conclude that some specific tasks do not benefit from generic language models pre-trained on large amounts of data. We explore the use of smaller architectures in domain-specific sub-languages and their benefits for French legal text. We prove that domain-specific pre-trained models can perform better than their equivalent generalised ones in the legal domain. Finally, we release JuriBERT, a new set of BERT models adapted to the French legal domain.