mlsanityData to 7 Oct 2026

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts

arXiv 2407.00256 ↗First posted 28 Jun 2024cs.AI cs.CL cs.LGcited by 16

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between in-context learning and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. The resulting method, codenamed Mixture-of-Prompts (MoP), achieves an average win rate of 81% against prior arts across several major benchmarks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 41 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

30 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
17 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2023Large Language Models as Optimizerscited by 830
  7. 2016Language Modeling with Gated Convolutional Networkscited by 688
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  10. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  12. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  13. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  14. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  15. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  16. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  17. 2022GPS: Genetic Prompt Search for Efficient Few-shot Learningcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  2. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  3. 2024QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learningcited by 10
  4. 2024PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creationcited by 9
  5. 2025A Survey of Automatic Prompt Optimization with Instruction-focused Heuristic-based Search Algorithmcited by 6
  6. 2025MIXRAG : Mixture-of-Experts Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 6
  7. 2025Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimizationcited by 5
  8. 2024BrainMAP: Learning Multiple Activation Pathways in Brain Networkscited by 2
  9. 2025Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoningcited by 2
  10. 2026Mixture of Demonstrations for Textual Graph Understanding and Question Answeringcited by 1
  11. 2026p1: Better Prompt Optimization with Fewer Promptscited by 1
  12. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
4 more
  1. 2026Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labelscited by 0
  2. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  3. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  4. 2026Adaptive-GEPA: Make Your Harness Fit Heterogeneous Requestscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  3. 2024Multi-expert Prompting Improves Reliability, Safety, and Usefulness of Large Language Models
  4. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  5. 2026HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
  6. 2024ConstitutionalExperts: Training a Mixture of Principle-based Prompts
  7. 2025Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities
  8. 2024Towards Generalist Prompting for Large Language Models by Mental Models