# NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension

arXiv: https://arxiv.org/abs/2603.05046 · HTML: https://mlsanity.com/paper/2603.05046/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 4 of 4)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Skywork: A More Open Bilingual Foundation Model](https://mlsanity.com/paper/2310.19341/) (2023)
- [Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages](https://mlsanity.com/paper/2508.17078/) (2025)

## Abstract (arXiv)

Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose NeuronMoE, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.