mlsanityData to 14 Oct 2026

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

arXiv 2604.25578 ↗First posted 28 Apr 2026cs.CL cs.AIcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-textscInstruct variants, which surpass the performance of competing models possessing 3--14× more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 59 of its 113 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

59 of 113 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models23 Jul
  2. 2025Qwen3 Technical Report14 May
  3. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  4. 2024Mixtral of Experts8 Jan
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2016Language Modeling with Gated Convolutional Networkscited by 702
  12. 2025Kimi K2: Open Agentic Intelligencecited by 567
  13. 20242 OLMo 2 Furiouscited by 340
  14. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  15. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  16. 2023The False Promise of Imitating Proprietary LLMscited by 263
  17. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  18. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  19. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  20. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  21. 2025Olmo 3cited by 129
  22. 2026Ministral 3cited by 114
  23. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  24. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  25. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  26. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 60
  27. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  28. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  29. 2025LFM2 Technical Reportcited by 37
  30. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  31. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  32. 2025OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Trainingcited by 21
  33. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  34. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  35. 2025Apertus: Democratizing Open and Compliant LLMs for Global Language Environmentscited by 12
  36. 2026Arcee Trinity Large Technical Reportcited by 11
  37. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 10
  38. 2026EuroLLM-22B: Technical Reportcited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Instella-MoE Technical Reportcited by 0
  2. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0
  3. 2026Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss Pairscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  2. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  4. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  5. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  6. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  7. 2024OLMoE: Open Mixture-of-Experts Language Models
  8. 2025Mixture of Experts in Large Language Models