# AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With

arXiv: https://arxiv.org/abs/2104.04052 · HTML: https://mlsanity.com/paper/2104.04052/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?](https://mlsanity.com/paper/2005.01330/) (2020)
- [Neural Modeling for Named Entities and Morphology (NEMO^2)](https://mlsanity.com/paper/2007.15620/) (2020)
- [What's Wrong with Hebrew NLP? And How to Make it Right](https://mlsanity.com/paper/1908.05453/) (2019)
- [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://mlsanity.com/paper/2102.01909/) (2021)

## Built on by (most cited first, 28 of 28)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew](https://mlsanity.com/paper/2308.16687/) (2023)
- [Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://mlsanity.com/paper/2211.15199/) (2022)
- [Breaking Character: Are Subwords Good Enough for MRLs After All?](https://mlsanity.com/paper/2204.04748/) (2022)
- [ParaShoot: A Hebrew Question Answering Dataset](https://mlsanity.com/paper/2109.11314/) (2021)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://mlsanity.com/paper/2304.11077/) (2023)
- [Offensive Hebrew Corpus and Detection using BERT](https://mlsanity.com/paper/2309.02724/) (2023)
- [Detecting Suicide Risk in Online Counseling Services: A Study in a Low-Resource Language](https://mlsanity.com/paper/2209.04830/) (2022)
- [ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development](https://mlsanity.com/paper/2307.08720/) (2023)
- [A Truly Joint Neural Architecture for Segmentation and Parsing](https://mlsanity.com/paper/2402.02564/) (2024)
- [Automated Extraction of Sentencing Decisions from Court Cases in the Hebrew Language](https://mlsanity.com/paper/2110.12383/) (2021)
- [Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language](https://mlsanity.com/paper/2208.01875/) (2022)
- [MRL Parsing Without Tears: The Case of Hebrew](https://mlsanity.com/paper/2403.06970/) (2024)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction](https://mlsanity.com/paper/2402.13906/) (2024)
- [A Language Modeling Approach to Diacritic-Free Hebrew TTS](https://mlsanity.com/paper/2407.12206/) (2024)
- [NLP meets psychotherapy: Using predicted client emotions and self-reported client emotions to measure emotional coherence](https://mlsanity.com/paper/2211.12512/) (2022)
- [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://mlsanity.com/paper/2405.07099/) (2024)
- [Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark](https://mlsanity.com/paper/2506.24117/) (2025)
- [HalleluBERT: Let Every Token That Has Meaning Bear Its Weight](https://mlsanity.com/paper/2510.21372/) (2025)
- [Unveiling Affective Polarization Trends in Parliamentary Proceedings](https://mlsanity.com/paper/2512.05231/) (2025)
- [Word-Embeddings Distinguish Denominal and Root-Derived Verbs in Semitic](https://mlsanity.com/paper/2208.05721/) (2022)
- [A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry](https://mlsanity.com/paper/2402.17371/) (2024)
- [Institutional-Level Monitoring of Immune Checkpoint Inhibitor IrAEs Using a Novel Natural Language Processing Algorithmic Pipeline](https://mlsanity.com/paper/2403.09708/) (2024)
- [Framework of Voting Prediction of Parliament Members](https://mlsanity.com/paper/2505.12535/) (2025)
- [CARE: Counselor-Aligned Response Engine for Online Mental-Health Support](https://mlsanity.com/paper/2604.21352/) (2026)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://mlsanity.com/paper/2606.19638/) (2026)
- [Same violence, different answer: how AI responds to coercive control against women across languages](https://mlsanity.com/paper/2608.01436/) (2026)

## Abstract (arXiv)

Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology and lie at the heart of many artificial intelligence advances. While advances reported for English using PLMs are unprecedented, reported advances using PLMs in Hebrew are few and far between. The problem is twofold. First, Hebrew resources available for training NLP models are not at the same order of magnitude as their English counterparts. Second, there are no accepted tasks and benchmarks to evaluate the progress of Hebrew PLMs on. In this work we aim to remedy both aspects. First, we present AlephBERT, a large pre-trained language model for Modern Hebrew, which is trained on larger vocabulary and a larger dataset than any Hebrew PLM before. Second, using AlephBERT we present new state-of-the-art results on multiple Hebrew tasks and benchmarks, including: Segmentation, Part-of-Speech Tagging, full Morphological Tagging, Named-Entity Recognition and Sentiment Analysis. We make our AlephBERT model publicly available, providing a single point of entry for the development of Hebrew NLP applications.