mlsanityData to 7 Oct 2026

HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

arXiv 2605.11255 ↗First posted 11 May 2026cs.CLcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Hebatron, a Hebrew-specialized open-weight large language model built on the NVIDIA Nemotron-3 sparse Mixture-of-Experts architecture. Training employs a three-phase easy-to-hard curriculum with continuous anti-forgetting anchoring, followed by supervised fine-tuning on 2 million bilingual Hebrew--English samples. The curriculum ordering alone yields a 3-point aggregate benchmark gain over the reversed configuration. Hebatron achieves a Hebrew reasoning average of 73.8%, outperforming DictaLM-3.0-24B-Thinking (68.9%) and remaining competitive with Gemma-3-27B-IT on GSM8K-HE and Israeli Trivia, while activating only 3B parameters per forward pass across a 30B-parameter model, delivering approximately 9 times higher inference throughput at native context lengths up to 65,536 tokens. To our knowledge, this is the first language-specific adaptation of the Nemotron-3 architecture for any target language, and the first open-weight Hebrew-specialized MoE model with native long-context support. Model weights are released openly to support further research in Hebrew and Semitic-language NLP.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 47 of its 62 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

47 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mixtral of Experts8 Jan

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
34 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  13. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  14. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  15. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  16. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  17. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  18. 2018An Empirical Model of Large-Batch Trainingcited by 268
  19. 2022FP8 Formats for Deep Learningcited by 187
  20. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  21. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  22. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  23. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  24. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  25. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  26. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  27. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  28. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  29. 2024Hermes 3 Technical Reportcited by 33
  30. 2021Multilingual Unsupervised Neural Machine Translation with Denoising Adapterscited by 22
  31. 2025Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Datacited by 16
  32. 2024Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilitiescited by 13
  33. 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 5
  34. 2026Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mixture of Experts for Low-Resource LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  3. 2025Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  5. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  6. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  7. 2025HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning
  8. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models