THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (e.g., domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-k~citeshazeer2017 and Top-p~citehuang-etal-2024-harder routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-p~citehuang-etal-2024-harder routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22% activated parameters on multi-domain translation tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 42 references that are in the map. First posted on arXiv 20 May 2025.
Builds on
31 of 42 references.
Closest ancestors · 9
Same direction, newest first
- 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free14 Oct
- 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling20 Aug
- 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts20 Feb
- 2023Task-Based MoE for Multitask Multilingual Machine Translation30 Aug
- 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation15 Dec
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
- 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan
Other references · 22
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2020Beyond English-Centric Multilingual Machine Translationcited by 561
- 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
- 2017Six Challenges for Neural Machine Translationcited by 335
- 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
- 2019Massively Multilingual Neural Machine Translationcited by 246
- 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
10 more
- 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
- 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
- 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
- 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
- 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 46
- 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
- 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
- 2020Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translationcited by 17
- 2023Learning Language-Specific Layers for Multilingual Machine Translationcited by 13
- 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
- 2025Mixture of Experts in Large Language Models
- 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
- 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
- 2024A Closer Look into Mixture-of-Experts in Large Language Models
- 2024Multi-Head Mixture-of-Experts
- 2026DOT-MoE: Differentiable Optimal Transport for MoEfication
- 2024Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation