# Efficient Ensemble Selection from Binary and Pairwise Feedback

arXiv: https://arxiv.org/abs/2605.09588 · HTML: https://mlsanity.com/paper/2605.09588/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 62 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 7 of 7)

- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [A Survey on Large Language Model Benchmarks](https://mlsanity.com/paper/2508.15361/) (2025)
- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [Diverse Committees with Incomplete or Inaccurate Approval Ballots](https://mlsanity.com/paper/2506.10843/) (2025)
- [Query-Based Committee Selection](https://mlsanity.com/paper/2603.29729/) (2026)
- [Learning Unanimously Acceptable Lotteries via Queries](https://mlsanity.com/paper/2604.17505/) (2026)

## Abstract (arXiv)

Organizations increasingly deploy multiple AI systems across task domains, but selecting a small, high-performing ensemble can require costly model calls, benchmark runs, and human evaluation. We study this selection problem as a distributional variant of multiwinner voting: tasks are drawn from an unknown domain distribution, each task induces feedback over candidate experts, and a committee's value on a task is determined by its best-performing member. We analyze both binary feedback, for tasks with correct/incorrect outcomes, and pairwise feedback, for tasks where candidate outputs are compared by preference. In the binary setting, the induced objective is coverage. We give exhaustive-elicitation baselines and matching worst-case query lower bounds, and we design a failure-conditioned greedy algorithm that preserves the standard (1-1/e) guarantee while obtaining instance-dependent query savings. In the pairwise setting, we study θ-winning committees. We show that full-information optimization admits a PTAS but no EPTAS under Gap-ETH, and that the objective is monotone but not submodular. This motivates a weighted ordinal coverage relaxation, which is submodular and supports a failure-conditioned greedy oracle under pairwise feedback. We then convert this oracle back into θ-type guarantees through finite-family auditing or a minimax wrapper. We also provide small-scale LLM experiments illustrating the predicted query savings and the role of complementarity in committee selection.