# DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew

arXiv: https://arxiv.org/abs/2308.16687 · HTML: https://mlsanity.com/paper/2308.16687/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Fast WordPiece Tokenization](https://mlsanity.com/paper/2012.15524/) (2020)
- [AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With](https://mlsanity.com/paper/2104.04052/) (2021)
- [Neural Modeling for Named Entities and Morphology (NEMO^2)](https://mlsanity.com/paper/2007.15620/) (2020)
- [Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://mlsanity.com/paper/2211.15199/) (2022)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://mlsanity.com/paper/2304.11077/) (2023)
- [A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsing](https://mlsanity.com/paper/2210.07873/) (2022)
- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://mlsanity.com/paper/2212.09682/) (2022)

## Built on by (most cited first, 17 of 17)

- [HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew](https://mlsanity.com/paper/2406.03897/) (2024)
- [Splintering Nonconcatenative Languages for Better Tokenization](https://mlsanity.com/paper/2503.14433/) (2025)
- [The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings](https://mlsanity.com/paper/2405.18115/) (2024)
- [MRL Parsing Without Tears: The Case of Hebrew](https://mlsanity.com/paper/2403.06970/) (2024)
- [Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization](https://mlsanity.com/paper/2507.08342/) (2025)
- [Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible](https://mlsanity.com/paper/2510.07178/) (2025)
- [Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings](https://mlsanity.com/paper/2407.20581/) (2024)
- [HebID: Detecting Social Identities in Hebrew-language Political Text](https://mlsanity.com/paper/2508.15483/) (2025)
- [The Enemy from Within: A Study of Political Delegitimization Discourse in Israeli Political Speech](https://mlsanity.com/paper/2508.15524/) (2025)
- [NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew](https://mlsanity.com/paper/2510.20386/) (2025)
- [Unveiling Affective Polarization Trends in Parliamentary Proceedings](https://mlsanity.com/paper/2512.05231/) (2025)
- [JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew](https://mlsanity.com/paper/2604.18041/) (2026)
- [ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2606.20179/) (2026)
- [Phonikud: Overcoming Phonetic Underspecification for Hebrew Text-To-Speech](https://mlsanity.com/paper/2506.12311/) (2025)
- [An Annotation Scheme for Factuality and its Application to Parliamentary Proceedings](https://mlsanity.com/paper/2509.26406/) (2025)
- [Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text](https://mlsanity.com/paper/2604.17108/) (2026)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://mlsanity.com/paper/2606.19638/) (2026)

## Abstract (arXiv)

We present DictaBERT, a new state-of-the-art pre-trained BERT model for modern Hebrew, outperforming existing models on most benchmarks. Additionally, we release three fine-tuned versions of the model, designed to perform three specific foundational tasks in the analysis of Hebrew texts: prefix segmentation, morphological tagging and question answering. These fine-tuned models allow any developer to perform prefix segmentation, morphological tagging and question answering of a Hebrew input with a single call to a HuggingFace model, without the need to integrate any additional libraries or code. In this paper we describe the details of the training as well and the results on the different benchmarks. We release the models to the community, along with sample code demonstrating their use. We release these models as part of our goal to help further research and development in Hebrew NLP.