mlsanityData to 14 Oct 2026

Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation

arXiv 2603.12577 ↗First posted 13 Mar 2026cs.CL cs.CVcited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks--where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 30 of its 44 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

30 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editing14 Nov
  2. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning25 Jan
  3. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality14 Oct
  4. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  5. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models20 Feb

Other references · 25

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
13 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2018Neural Network Acceptability Judgmentscited by 958
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  5. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  6. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  7. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  8. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  9. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  10. 2025Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspacecited by 11
  11. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  12. 2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
  13. 2025HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRAcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning
  2. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
  3. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  4. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
  5. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  6. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  7. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
  8. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation