mlsanityData to 14 Oct 2026

Mellum2 Technical Report

arXiv 2605.31268 ↗First posted 29 May 2026cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Mellum 2, an open-weight 12B-parameter Mixture-of-Experts (MoE) language model with 2.5B active parameters per token. Mellum 2 is a general-purpose language model specialized in software engineering, spanning code generation and editing, debugging, multi-step reasoning, tool use and function calling, agentic coding, and conversational programming assistance, and it is the successor to the completion-focused 4B dense Mellum model. The architecture builds on the Mixture-of-Experts (64 experts, 8 active) and combines Grouped-Query Attention with 4 KV heads, Sliding Window Attention on three of every four layers, and a single Multi-Token Prediction head that doubles as both an auxiliary pre-training objective and a built-in draft model for speculative decoding; each choice was validated by ablation with inference efficiency on commodity GPUs as a design constraint. Pre-training spans approximately 10.6 trillion tokens through a three-phase curriculum that progressively shifts the mixture from diverse web data toward curated code and mathematical content, optimized with Muon under FP8 hybrid precision and a Warmup-Hold-Decay schedule with linear decay to zero. The pre-trained base is extended to a 128K context window via a layer-selective YaRN and then post-trained in two stages (supervised fine-tuning followed by RLVR), yielding two released variants: an Instruct model that answers directly and a Thinking model that emits an explicit reasoning trace before its final answer. Across code generation, math and reasoning, tool use, knowledge, and safety benchmarks, Mellum 2 is competitive with open-weight baselines in the 4B-14B range while running at the per-token compute of a 2.5B dense model. We release the base, instruct, and thinking checkpoints, together with this report on the architecture decisions, data pipeline, and training recipe behind them, under the Apache 2.0 license.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 68 of its 78 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

68 of 78 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM22 Mar
  3. 2024Qwen2.5 Technical Report19 Dec
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  5. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 61

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2023Mistral 7Bcited by 3,164
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2025Gemma 3 Technical Reportcited by 1,824
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2020GLU Variants Improve Transformercited by 1,804
  13. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  14. 2019Root Mean Square Layer Normalizationcited by 1,480
  15. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  16. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  17. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  18. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  19. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  20. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  21. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  22. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  23. 2023StarCoder: may the source be with you!cited by 799
  24. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  25. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  26. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  27. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  28. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  29. 2025Muon is Scalable for LLM Trainingcited by 428
  30. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  31. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  32. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  33. 2020Query-Key Normalization for Transformerscited by 280
  34. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  35. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  36. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  37. 2022FP8 Formats for Deep Learningcited by 196
  38. 2024Are We Done with MMLU?cited by 194

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  2. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  5. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  6. 2024Mixture of Diverse Size Experts
  7. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  8. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models