# Wisdom and Delusion of LLM Ensembles for Code Generation and Repair

arXiv: https://arxiv.org/abs/2510.21513 · HTML: https://mlsanity.com/paper/2510.21513/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 24 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023)
- [The Larger the Better? Improved LLM Code-Generation via Budget Reallocation](https://mlsanity.com/paper/2404.00725/) (2024)
- [Trust Enhancement Issues in Program Repair](https://mlsanity.com/paper/2108.13064/) (2021)
- [How Far Can We Go with Practical Function-Level Program Repair?](https://mlsanity.com/paper/2404.12833/) (2024)
- [Large Language Models for Code Generation: The Practitioners Perspective](https://mlsanity.com/paper/2501.16998/) (2025)
- [Ensemble Learning for Large Language Models in Text and Code Generation: A Survey](https://mlsanity.com/paper/2503.13505/) (2025)
- [The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models](https://mlsanity.com/paper/2505.02931/) (2025)
- [EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Code](https://mlsanity.com/paper/2411.16561/) (2024)
- [Practical Program Repair via Preference-based Ensemble Strategy](https://mlsanity.com/paper/2309.08211/) (2023)
- [Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection](https://mlsanity.com/paper/2503.23448/) (2025)
- [Ensembling Large Language Models for Code Vulnerability Detection: An Empirical Evaluation](https://mlsanity.com/paper/2509.12629/) (2025)

## Built on by (most cited first, 3 of 3)

- [Multi-LLM Collaboration for Medication Recommendation](https://mlsanity.com/paper/2512.05066/) (2025)
- [Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration](https://mlsanity.com/paper/2602.04291/) (2026)
- [tap: A File-Based Protocol for Heterogeneous LLM Agent Collaboration](https://mlsanity.com/paper/2606.14445/) (2026)

## Abstract (arXiv)

Today's pursuit of a single Large Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.