# AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach

arXiv: https://arxiv.org/abs/2410.10896 · HTML: https://mlsanity.com/paper/2410.10896/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts.
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Placed in

- Mixture-of-experts / [Mixture of LoRA experts](https://mlsanity.com/moe/mixture-of-lora-experts/)

## Builds on (most cited first, 11 of 11)

- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning](https://mlsanity.com/paper/2312.12379/) (2023)
- [Higher Layers Need More LoRA Experts](https://mlsanity.com/paper/2402.08562/) (2024)
- [Learning to Route Among Specialized Experts for Zero-Shot Generalization](https://mlsanity.com/paper/2402.05859/) (2024)
- [MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning](https://mlsanity.com/paper/2402.11260/) (2024)
- [Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization](https://mlsanity.com/paper/2311.09344/) (2023)
- [CCoE: A Compact and Efficient LLM Framework with Multi-Expert Collaboration for Resource-Limited Settings](https://mlsanity.com/paper/2407.11686/) (2024)

## Built on by (most cited first, 1 of 1)

- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)

## Abstract (arXiv)

The advent of Large Language Models (LLMs) has ushered in a new era of artificial intelligence, with the potential to transform various sectors through automation and insightful analysis. The Mixture of Experts (MoE) architecture has been proposed as a solution to enhance model performance in complex tasks. Yet, existing MoE models struggle with task-specific learning and interpretability, especially in fields like medicine where precision is critical. This paper introduces the Adaptive Task-planing Mixture of Experts(AT-MoE), an innovative architecture designed to address these limitations. We first train task-specific experts via LoRA approach to enhance problem-solving capabilities and interpretability in specialized areas. Subsequently, we introduce a layer-wise adaptive grouped routing module that optimizes module fusion based on complex task instructions, ensuring optimal task resolution. The grouped routing module first perform overall weight allocation from the dimension of the expert group, and then conduct local weight normalization adjustments within the group. This design maintains multi-dimensional balance, controllability, and interpretability, while facilitating task-specific fusion in response to complex instructions.