mlsanityData to 7 Oct 2026

Boosted Prompt Ensembles for Large Language Models

arXiv 2304.05970 ↗First posted 12 Apr 2023cs.CL cs.LGcited by 36

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Methods such as chain-of-thought prompting and self-consistency have pushed the frontier of language model reasoning performance with no additional training. To further improve performance, we propose a prompt ensembling method for large language models, which uses a small dataset to construct a set of few shot prompts that together comprise a ``boosted prompt ensemble''. The few shot examples for each prompt are chosen in a stepwise fashion to be ``hard'' examples on which the previous step's ensemble is uncertain. We show that this outperforms single-prompt output-space ensembles and bagged prompt-space ensembles on the GSM8k and AQuA datasets, among others. We propose both train-time and test-time versions of boosted prompting that use different levels of available annotation and conduct a detailed empirical study of our algorithm.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 92 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

36 of 92 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters20 Dec
  2. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  3. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  4. 2022Large Language Models are Zero-Shot Reasoners24 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
18 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  13. 2018AI safety via debatecited by 384
  14. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  15. 2021Ensemble deep learning: A reviewcited by 283
  16. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  17. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  18. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 12; 2025: 8; 2026: 5.

36 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  3. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  4. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  5. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  7. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  8. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
  9. 2024Smoothie: Label Free Language Model Routingcited by 31
  10. 2023PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refinecited by 28
  11. 2023LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMscited by 27
  12. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
24 more
  1. 2023Mathematical Language Models: A Surveycited by 24
  2. 2024Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Modelscited by 22
  3. 2024Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challengescited by 21
  4. 2024Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generationcited by 20
  5. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  6. 2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critiquecited by 13
  7. 2024Rethinking VLMs and LLMs for Image Classificationcited by 12
  8. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  9. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  10. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  11. 2024Eyeballing Combinatorial Problems: A Case Study of Using Multimodal Large Language Models to Solve Traveling Salesman Problemscited by 8
  12. 2024SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Frameworkcited by 3
  13. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  14. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 3
  15. 2024Translate-and-Revise: Boosting Large Language Models for Constrained Translationcited by 2
  16. 2025Can AI automatically analyze public opinion? A LLM agents-based agentic pipeline for timely public opinion analysiscited by 2
  17. 2025Reasoning-Intensive Regressioncited by 2
  18. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloadingcited by 2
  19. 2023Leveraging Few-Shot Data Augmentation and Waterfall Prompting for Response Generationcited by 1
  20. 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Frameworkcited by 1
  21. 2025PENEX: AdaBoost-Inspired Neural Network Regularizationcited by 0
  22. 2026Residual Skill Optimization for Text-to-SQL Ensemblescited by 0
  23. 2026Data-Efficient Adaptation of LLMs via Attention Head Reweightingcited by 0
  24. 2026Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Computecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks
  2. 2023Batch Prompting: Efficient Inference with Large Language Model APIs
  3. 2026TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregation
  4. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  5. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
  6. 2025Self-Supervised Prompt Optimization
  7. 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
  8. 2024Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions