mlsanityData to 7 Oct 2026

AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach

arXiv 2410.10896 ↗First posted 12 Oct 2024cs.LG cs.AI cs.CEcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

The advent of Large Language Models (LLMs) has ushered in a new era of artificial intelligence, with the potential to transform various sectors through automation and insightful analysis. The Mixture of Experts (MoE) architecture has been proposed as a solution to enhance model performance in complex tasks. Yet, existing MoE models struggle with task-specific learning and interpretability, especially in fields like medicine where precision is critical. This paper introduces the Adaptive Task-planing Mixture of Experts(AT-MoE), an innovative architecture designed to address these limitations. We first train task-specific experts via LoRA approach to enhance problem-solving capabilities and interpretability in specialized areas. Subsequently, we introduce a layer-wise adaptive grouped routing module that optimizes module fusion based on complex task instructions, ensuring optimal task resolution. The grouped routing module first perform overall weight allocation from the dimension of the expert group, and then conduct local weight normalization adjustments within the group. This design maintains multi-dimensional balance, controllability, and interpretability, while facilitating task-specific fusion in response to complex instructions.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 11 of its 16 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

11 of 16 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Higher Layers Need More LoRA Experts13 Feb
  2. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec

Other references · 9

  1. 2024Mixtral of Expertscited by 1,785
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  5. 2024Mixture of LoRA Expertscited by 179
  6. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  7. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  8. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  9. 2024CCoE: A Compact and Efficient LLM Framework with Multi-Expert Collaboration for Resource-Limited Settingscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Mixture of Experts in Large Language Modelscited by 15

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  2. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models
  3. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
  4. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
  5. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
  6. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  7. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  8. 2023Improving Expert Specialization in Mixture of Experts