# Hebrew Diacritics Restoration using Visual Representation

arXiv: https://arxiv.org/abs/2510.26521 · HTML: https://mlsanity.com/paper/2510.26521/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Text Rendering Strategies for Pixel Language Models](https://mlsanity.com/paper/2311.00522/) (2023)
- [What's Wrong with Hebrew NLP? And How to Make it Right](https://mlsanity.com/paper/1908.05453/) (2019)
- [PIXAR: Auto-Regressive Language Modeling in Pixel Space](https://mlsanity.com/paper/2401.03321/) (2024)
- [Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://mlsanity.com/paper/2211.15199/) (2022)
- [On the Influence of Masking Policies in Intermediate Pre-training](https://mlsanity.com/paper/2104.08840/) (2021)
- [Restoring Hebrew Diacritics Without a Dictionary](https://mlsanity.com/paper/2105.05209/) (2021)
- [Evaluating Pixel Language Models on Non-Standardized Languages](https://mlsanity.com/paper/2412.09084/) (2024)
- [Automatic Restoration of Diacritics for Speech Data Sets](https://mlsanity.com/paper/2311.10771/) (2023)
- [Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization](https://mlsanity.com/paper/2406.05760/) (2024)
- [Don't Touch My Diacritics](https://mlsanity.com/paper/2410.24140/) (2024)
- [Diacritics Restoration using BERT with Analysis on Czech language](https://mlsanity.com/paper/2105.11408/) (2021)
- [Splintering Nonconcatenative Languages for Better Tokenization](https://mlsanity.com/paper/2503.14433/) (2025)
- [Nakdan: Professional Hebrew Diacritizer](https://mlsanity.com/paper/2005.03312/) (2020)
- [PHD: Pixel-Based Language Modeling of Historical Documents](https://mlsanity.com/paper/2310.18343/) (2023)
- [Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew](https://mlsanity.com/paper/2311.00658/) (2023)
- [Attention-Guided Masked Autoencoders For Learning Image Representations](https://mlsanity.com/paper/2402.15172/) (2024)
- [Interplay of Machine Translation, Diacritics, and Diacritization](https://mlsanity.com/paper/2404.05943/) (2024)

## Abstract (arXiv)

Diacritics restoration in Hebrew is a fundamental task for ensuring accurate word pronunciation and disambiguating textual meaning. Despite the language's high degree of ambiguity when unvocalized, recent machine learning approaches have significantly advanced performance on this task. In this work, we present DiVRit, a novel system for Hebrew diacritization that frames the task as a zero-shot classification problem. Our approach operates at the word level, selecting the most appropriate diacritization pattern for each undiacritized word from a dynamically generated candidate set, conditioned on the surrounding textual context. A key innovation of DiVRit is its use of a Hebrew Visual Language Model to process diacritized candidates as images, allowing diacritic information to be embedded directly within their vector representations while the surrounding context remains tokenization-based. Through a comprehensive evaluation across various configurations, we demonstrate that the system effectively performs diacritization without relying on complex, explicit linguistic analysis. Notably, in an ``oracle'' setting where the correct diacritized form is guaranteed to be among the provided candidates, DiVRit achieves a high level of accuracy. Furthermore, strategic architectural enhancements and optimized training methodologies yield significant improvements in the system's overall generalization capabilities. These findings highlight the promising potential of visual representations for accurate and automated Hebrew diacritization.