mlsanityData to 7 Oct 2026

HUKUKBERT: Domain-Specific Language Model for Turkish Law

arXiv 2604.04790 ↗First posted 6 Apr 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural language processing (NLP) advances have powered a generation of LegalTech systems, but Turkish law remains under-served by domain-specific data and models. English has legal encoders such as LEGAL-BERT; no comparable high-volume Turkish counterpart exists. We introduce HukukBERT, a Turkish legal language model trained on a 19 GB cleaned corpus using a hybrid domain-adaptive pre-training (DAPT) recipe that mixes Whole-Word Masking, Token Span Masking, Word Span Masking, and targeted Keyword Masking. We compared our 48K WordPiece tokenizer and DAPT pipeline against general-purpose and existing domain-specific Turkish models. On the Legal Cloze Test - a masked legal term prediction benchmark over Turkish court decisions - HukukBERT reaches 84.40% Top-1 accuracy and beats every baseline we tested. The Legal Cloze Test is synthetically constructed, so its passages are absent from the pre-training corpus by construction, eliminating train-test contamination. On the downstream task of structural segmentation of official Turkish court decisions, it reaches a 92.8% document pass rate. We release HukukBERT to support Turkish legal NLP work in named entity recognition, judgment prediction, and document classification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

18 of 23 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  11. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  12. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
6 more
  1. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  2. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  3. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  4. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  5. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  6. 2024LegalTurk Optimized BERT for Multi-Label Text Classification and NERcited by 1

Built on by

Not cited yet.