mlsanityData to 14 Oct 2026

HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts

arXiv 2608.29790 ↗First posted 30 Aug 2026cs.CLcited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 32 of its 45 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

32 of 45 references.

References · 32

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  10. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  11. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  12. 2016NewsQA: A Machine Comprehension Datasetcited by 331
20 more
  1. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  2. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  3. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  4. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184
  5. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  6. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  7. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  8. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  9. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  10. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  11. 2018DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehensioncited by 71
  12. 2017Identifying beneficial task relations for multi-task learning in deep neural networkscited by 59
  13. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  14. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  15. 2023Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptationcited by 12
  16. 2023One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learningcited by 8
  17. 2025Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuningcited by 6
  18. 2025PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuningcited by 6
  19. 2023When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLPcited by 5
  20. 2023MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehensioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models
  2. 2026Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
  3. 2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
  4. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
  5. 2025PromptFlow: Training Prompts Like Neural Networks
  6. 2025A Survey on Prompt Tuning
  7. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  8. 2026Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis