# MenakBERT -- Hebrew Diacriticizer

arXiv: https://arxiv.org/abs/2410.02417 · HTML: https://mlsanity.com/paper/2410.02417/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Breaking Character: Are Subwords Good Enough for MRLs After All?](https://mlsanity.com/paper/2204.04748/) (2022)
- [Restoring Hebrew Diacritics Without a Dictionary](https://mlsanity.com/paper/2105.05209/) (2021)

## Abstract (arXiv)

Diacritical marks in the Hebrew language give words their vocalized form. The task of adding diacritical marks to plain Hebrew text is still dominated by a system that relies heavily on human-curated resources. Recent models trained on diacritized Hebrew texts still present a gap in performance. We use a recently developed char-based PLM to narrowly bridge this gap. Presenting MenakBERT, a character level transformer pretrained on Hebrew text and fine-tuned to produce diacritical marks for Hebrew sentences. We continue to show how finetuning a model for diacritizing transfers to a task such as part of speech tagging.