mlsanityData to 14 Oct 2026

Multi-expert Prompting Improves Reliability, Safety, and Usefulness of Large Language Models

arXiv 2411.00492 ↗First posted 1 Nov 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We present Multi-expert Prompting, a novel enhancement of ExpertPrompting (Xu et al., 2023), designed to improve the large language model (LLM) generation. Specifically, it guides an LLM to fulfill an input instruction by simulating multiple experts, aggregating their responses, and selecting the best among individual and aggregated responses. This process is performed in a single chain of thoughts through our seven carefully designed subtasks derived from the Nominal Group Technique (Ven and Delbecq, 1974), a well-established decision-making framework. Our evaluations demonstrate that Multi-expert Prompting significantly outperforms ExpertPrompting and comparable baselines in enhancing the truthfulness, factuality, informativeness, and usefulness of responses while reducing toxicity and hurtfulness. It further achieves state-of-the-art truthfulness by outperforming the best baseline by 8.69% with ChatGPT. Multi-expert Prompting is efficient, explainable, and highly adaptable to diverse scenarios, eliminating the need for manual prompt construction.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

34 of 60 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
22 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  8. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  9. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  10. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  11. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 142
  12. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  13. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  14. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  15. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  16. 2024From Role-Play to Drama-Interaction: An LLM Solutioncited by 42
  17. 2021N24News: A New Dataset for Multimodal News Classificationcited by 40
  18. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  19. 2016Echo Chambers: Emotional Contagion and Group Polarization on Facebookcited by 27
  20. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 25
  21. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 22
  22. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  2. 2025A Self-Improving Coding Agentcited by 79
  3. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 9
  4. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  5. 2025Auspex: Building Threat Modeling Tradecraft into an Artificial Intelligence-based Copilotcited by 3
  6. 2026Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loopscited by 3
  7. 2026From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modelingcited by 3
  8. 2025SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIscited by 1
  9. 2025Reasoning-Aware Prompt Orchestration: A Foundation Model for Multi-Agent Language Model Coordinationcited by 1
  10. 2026SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidancecited by 1
  11. 2025Basic Category Usage in Vision Language Modelscited by 0
  12. 2026CatalogAgent: A Supervisor-mediated Self-Learning System Enabling Context Engineering for GenAI Modelscited by 0
2 more
  1. 2026Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuningcited by 0
  2. 2026Reliable Fusion of Conflicting Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Generalist Prompting for Large Language Models by Mental Models
  2. 2026ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debate
  4. 2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
  5. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
  6. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  7. 2024Think Beyond Size: Adaptive Prompting for More Effective Reasoning
  8. 2023Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing