mlsanityData to 14 Oct 2026

KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications

arXiv 2503.17247 ↗First posted 21 Mar 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We present the KL3M tokenizers, a family of specialized tokenizers for legal, financial, and governmental text. Despite established work on tokenization, specialized tokenizers for professional domains remain understudied. Our paper offers two main contributions to this area. First, we introduce domain-specific BPE tokenizers for legal, financial, and governmental text. Our kl3m-004-128k-cased tokenizer uses 9-17% fewer tokens than GPT-4o and Llama3 for domain-specific documents, despite having a smaller vocabulary. For specialized terminology, our cased tokenizer is even more efficient, using up to 83% fewer tokens for legal terms and 39% fewer tokens for financial terms. Second, we develop character-level BPE tokenizers (4K, 8K, and 16K vocabulary sizes) for text correction tasks like OCR post-processing. These tokenizers keep consistent token boundaries between error-containing and correct text, making it easier for models to learn correction patterns. These tokenizers help professional applications by fitting more text in context windows, reducing computational needs, and preserving the meaning of domain-specific terms. Our analysis shows these efficiency gains directly benefit the processing of long legal and financial documents. We release all tokenizers and code through GitHub and Hugging Face to support further research in specialized tokenization.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

15 of 20 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  8. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  9. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  12. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
3 more
  1. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  2. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  3. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 6
  2. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 6
  3. 2025ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddingscited by 5
  4. 2025GRAPH-GRPO-LEX: Contract Graph Modeling and Reinforcement Learning with Group Relative Policy Optimizationcited by 1