# Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

arXiv: https://arxiv.org/abs/2309.14568 · HTML: https://mlsanity.com/paper/2309.14568/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Neural Modeling for Named Entities and Morphology (NEMO^2)](https://mlsanity.com/paper/2007.15620/) (2020)
- [ParaShoot: A Hebrew Question Answering Dataset](https://mlsanity.com/paper/2109.11314/) (2021)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://mlsanity.com/paper/2304.11077/) (2023)
- [Improving Low Compute Language Modeling with In-Domain Embedding Initialisation](https://mlsanity.com/paper/2009.14109/) (2020)

## Built on by (most cited first, 1 of 1)

- [Same violence, different answer: how AI responds to coercive control against women across languages](https://mlsanity.com/paper/2608.01436/) (2026)

## Abstract (arXiv)

We present DictaLM, a large-scale language model tailored for Modern Hebrew. Boasting 7B parameters, this model is predominantly trained on Hebrew-centric data. As a commitment to promoting research and development in the Hebrew language, we release both the foundation model and the instruct-tuned model under a Creative Commons license. Concurrently, we introduce DictaLM-Rab, another foundation model geared towards Rabbinic/Historical Hebrew. These foundation models serve as ideal starting points for fine-tuning various Hebrew-specific tasks, such as instruction, Q&A, sentiment analysis, and more. This release represents a preliminary step, offering an initial Hebrew LLM model for the Hebrew NLP community to experiment with.