# Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities

arXiv: https://arxiv.org/abs/2407.07080 · HTML: https://mlsanity.com/paper/2407.07080/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Extrapolating Large Language Models to Non-English by Aligning Languages](https://mlsanity.com/paper/2308.04948/) (2023)
- [Efficiently Adapting Pretrained Language Models To New Languages](https://mlsanity.com/paper/2311.05741/) (2023)
- [RakutenAI-7B: Extending Large Language Models for Japanese](https://mlsanity.com/paper/2403.15484/) (2024)

## Built on by (most cited first, 13 of 13)

- [Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model](https://mlsanity.com/paper/2502.01691/) (2025)
- [HebID: Detecting Social Identities in Hebrew-language Political Text](https://mlsanity.com/paper/2508.15483/) (2025)
- [The Enemy from Within: A Study of Political Delegitimization Discourse in Israeli Political Speech](https://mlsanity.com/paper/2508.15524/) (2025)
- [NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew](https://mlsanity.com/paper/2510.20386/) (2025)
- [Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs](https://mlsanity.com/paper/2602.02104/) (2026)
- [Effective QA-driven Annotation of Predicate-Argument Relations Across Languages](https://mlsanity.com/paper/2602.22865/) (2026)
- [HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://mlsanity.com/paper/2605.11255/) (2026)
- [Mixture of Experts for Low-Resource LLMs](https://mlsanity.com/paper/2605.17598/) (2026)
- [Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://mlsanity.com/paper/2512.11502/) (2025)
- [A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs](https://mlsanity.com/paper/2603.26236/) (2026)
- [Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text](https://mlsanity.com/paper/2604.17108/) (2026)
- [When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates](https://mlsanity.com/paper/2606.13218/) (2026)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://mlsanity.com/paper/2606.19638/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) in low-resource languages such as Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a substantial corpus of approximately 200 billion tokens in both Hebrew and English. Adapting a pre-trained model to a new language involves specialized techniques that differ significantly from training a model from scratch or further training existing models on well-resourced languages such as English. We outline these novel training methodologies, which facilitate effective learning and adaptation to the linguistic properties of Hebrew. Additionally, we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to enhance its performance on task-specific instructions. To rigorously evaluate our models, we introduce a new benchmark suite for Hebrew LLM evaluation, covering a diverse set of tasks including Question Answering, Sentiment Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work not only addresses the intricacies of training LLMs in low-resource languages but also proposes a framework that can be leveraged for adapting other LLMs to various non-English languages, contributing to the broader field of multilingual NLP.