# MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

arXiv: https://arxiv.org/abs/2608.05850 · HTML: https://mlsanity.com/paper/2608.05850/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2404.04042/) (2024)
- [SentAlign: Accurate and Scalable Sentence Alignment](https://mlsanity.com/paper/2311.08982/) (2023)
- [Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models](https://mlsanity.com/paper/2505.22232/) (2025)
- [Pretraining Language Models Using Translationese](https://mlsanity.com/paper/2403.13638/) (2024)
- [Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation](https://mlsanity.com/paper/2506.00288/) (2025)
- [Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting](https://mlsanity.com/paper/2503.01493/) (2025)
- [Pula: Training Large Language Models for Setswana](https://mlsanity.com/paper/2408.02239/) (2024)
- [A Part-of-Speech Tagger for Yiddish](https://mlsanity.com/paper/2204.01175/) (2022)
- [Jochre 3 and the Yiddish OCR corpus](https://mlsanity.com/paper/2501.08442/) (2025)

## Abstract (arXiv)

We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. Existing multilingual corpora and benchmarks are often poor proxies for the language, containing substantial amounts of noisy, machine-translated, and misclassified text. We address these gaps by introducing Oytser, a high-quality Yiddish pretraining corpus that combines contemporary web-native sources with literary materials, and Kashes, a multi-task benchmark spanning translation, linguistic analysis, information extraction, and language understanding. Using these resources, we continue pretraining Llama 3.1 8B to obtain MameLoshnLM. Across the tasks in the benchmark, MameLoshnLM outperforms open baselines of similar scale. Our analyses show that these gains are not only quantitative: relative to general-purpose multilingual models, MameLoshnLM better captures language-defining lexical and morphological patterns, pointing to a broader failure mode of noisy web-scale multilingual data for low-resource languages. Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.