# Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language

arXiv: https://arxiv.org/abs/2208.01875 · HTML: https://mlsanity.com/paper/2208.01875/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 8 references that are in the map.
- First posted on arXiv 3 Aug 2022.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With](https://mlsanity.com/paper/2104.04052/) (2021)
- [What's Wrong with Hebrew NLP? And How to Make it Right](https://mlsanity.com/paper/1908.05453/) (2019)
- [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://mlsanity.com/paper/2102.01909/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Statistical Exploration of Text Partition Into Constituents: The Case of the Priestly Source in the Books of Genesis and Exodus](https://mlsanity.com/paper/2305.02170/) (2023)
- [Style Classification of Rabbinic Literature for Detection of Lost Midrash Tanhuma Material](https://mlsanity.com/paper/2211.09710/) (2022)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://mlsanity.com/paper/2606.19638/) (2026)

## Abstract (arXiv)

We present a new pre-trained language model (PLM) for Rabbinic Hebrew, termed Berel (BERT Embeddings for Rabbinic-Encoded Language). Whilst other PLMs exist for processing Hebrew texts (e.g., HeBERT, AlephBert), they are all trained on modern Hebrew texts, which diverges substantially from Rabbinic Hebrew in terms of its lexicographical, morphological, syntactic and orthographic norms. We demonstrate the superiority of Berel on Rabbinic texts via a challenge set of Hebrew homographs. We release the new model and homograph challenge set for unrestricted use.