mlsanityData to 14 Oct 2026

Branch-Solve-Merge Improves Large Language Model Evaluation and Generation

arXiv 2310.15123 ↗First posted 23 Oct 2023cs.CL cs.AI cs.LGcited by 106

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) are frequently used for multi-faceted language generation and evaluation tasks that involve satisfying intricate user constraints or taking into account multiple aspects and criteria. However, their performance can fall short, due to the model's lack of coherence and inability to plan and decompose the problem. We propose Branch-Solve-Merge (BSM), a Large Language Model program (Schlag et al., 2023) for tackling such challenging natural language tasks. It consists of branch, solve, and merge modules that are parameterized with specific prompts to the base LLM. These three modules plan a decomposition of the task into multiple parallel sub-tasks, independently solve them, and fuse the solutions to the sub-tasks. We apply our method to the tasks of LLM response evaluation and constrained text generation and evaluate its effectiveness with multiple LLMs, including Vicuna, LLaMA-2-chat, and GPT-4. BSM improves the evaluation correctness and consistency for each LLM by enhancing human-LLM agreement by up to 26%, reducing length and pairwise position biases by up to 50%, and allowing LLaMA2-chat to match or outperform GPT-4 on most domains. On a constraint story generation task, BSM improves the coherence of stories while also improving constraint satisfaction by 12%.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

43 of 56 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
31 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2023LIMA: Less Is More for Alignmentcited by 1,100
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  15. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  16. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  17. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  18. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  20. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  21. 2022Language Model Cascadescited by 107
  22. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  23. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  24. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  25. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  26. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65
  27. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  28. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 54
  29. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  30. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  31. 2022MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 47; 2025: 33; 2026: 23.

106 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024Self-Rewarding Language Modelscited by 681
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  6. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  7. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  8. 2024Distilling System 2 into System 1cited by 159
  9. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  10. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  11. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  12. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 110
38 more
  1. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  2. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  3. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  4. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  5. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 73
  6. 2024Self-Taught Evaluatorscited by 72
  7. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  8. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 60
  9. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  10. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  11. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  12. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  13. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  14. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  15. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  16. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  17. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  18. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  19. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  20. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  21. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  22. 2025Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoningcited by 24
  23. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  24. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21
  25. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  26. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  27. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  28. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  29. 2024Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Coursecited by 19
  30. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 19
  31. 2023Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responsescited by 18
  32. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  33. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signalscited by 18
  34. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  35. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  36. 2024TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planningcited by 14
  37. 2024Small But Funny: A Feedback-Driven Approach to Humor Distillationcited by 12
  38. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12

50 most cited of 106 All 106 (JSON)