mlsanityData to 7 Oct 2026

Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

arXiv 2503.05641 ↗First posted 7 Mar 2025cs.CL cs.AI cs.LGcited by 24

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks. However, task-level expert selection is often too coarse-grained, since different instances may require different expertise. To address this, we propose Skill-MoE, a symbolic, skill-based, and gradient-free Mixture-of-Experts framework for instance-level expert selection. Skill-MoE infers skills (e.g., algebra in mathematics) from each query, selects experts based on skill relevance, and lets each expert generate its own reasoning. The resulting k outputs are then synthesized by an aggregator chosen for its ability to integrate diverse responses. While instance-level selection substantially improves performance, naively implementing it incurs heavy overhead from repeated model loading and offloading. We address this with a batch inference strategy that groups instances by assigned experts, allowing each model to be loaded only once. As a result, Skill-MoE integrates 16 expert models on a single GPU with runtime comparable to prior multi-agent baselines using 4 GPUs. Across diverse benchmarks (MMLU-Pro, GPQA, AIME, and MedMCQA), Skill-MoE achieves an average absolute improvement of 8.15% over the best baseline. It also generalizes well to unseen tasks and outperforms discussion-based methods without requiring expensive multi-round interactions.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 25 of its 67 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

25 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?2 Feb
  2. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoning22 Nov

Other references · 23

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  9. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
11 more
  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  4. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  5. 2022Large Language Models Are Reasoning Teacherscited by 318
  6. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  7. 2022Teaching Small Language Models to Reasoncited by 279
  8. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  9. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  10. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  11. 2021Towards More Effective and Economic Sparsely-Activated Modelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 6.

24 papers.

Sort:

By area:

  1. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  2. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 26
  3. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  4. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  5. 2025TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixturecited by 13
  6. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  7. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  8. 2025S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoningcited by 8
  9. 2025MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregationcited by 6
  10. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedbackcited by 6
  11. 2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
  12. 2025CTTS: Collective Test-Time Scalingcited by 3
12 more
  1. 2025When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensemblingcited by 3
  2. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesiscited by 3
  3. 2025MathEDU: Feedback Generation on Problem-Solving Processes for Mathematical Learning Supportcited by 2
  4. 2025Second Opinion Matters: Towards Adaptive Clinical AI via the Consensus of Expert Model Ensemblecited by 1
  5. 2026Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scalecited by 1
  6. 2025A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancementcited by 0
  7. 2025Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Makingcited by 0
  8. 2025Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvementcited by 0
  9. 2026PASM: Population Adaptive Symbolic Mixture-of-Experts Model for Cross-location Hurricane Evacuation Decision Predictioncited by 0
  10. 2026LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deploymentcited by 0
  11. 2026Skill Training with Corruption and Reconstruction Loopcited by 0
  12. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models
  2. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  3. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  4. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models
  6. 2026MoE Lens -- An Expert Is All You Need
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  8. 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts