# Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models

arXiv: https://arxiv.org/abs/2012.01988 · HTML: https://mlsanity.com/paper/2012.01988/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 30 of 39)

- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [Faster Cascades via Speculative Decoding](https://mlsanity.com/paper/2405.19261/) (2024)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Improved Feature Distillation via Projector Ensemble](https://mlsanity.com/paper/2210.15274/) (2022)
- [Online Cascade Learning for Efficient Inference over Streams](https://mlsanity.com/paper/2402.04513/) (2024)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Agreement-Based Cascading for Efficient Inference](https://mlsanity.com/paper/2407.02348/) (2024)
- [Cascade-Aware Training of Language Models](https://mlsanity.com/paper/2406.00060/) (2024)
- [Towards Inference Efficient Deep Ensemble Learning](https://mlsanity.com/paper/2301.12378/) (2023)
- [BabyBear: Cheap inference triage for expensive language models](https://mlsanity.com/paper/2205.11747/) (2022)
- [Semantic Agreement Enables Efficient Open-Ended LLM Cascades](https://mlsanity.com/paper/2509.21837/) (2025)
- [MapReader: A Computer Vision Pipeline for the Semantic Exploration of Maps at Scale](https://mlsanity.com/paper/2111.15592/) (2021)
- [CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2205.08012/) (2022)
- [Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification](https://mlsanity.com/paper/2302.00368/) (2023)
- [Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach](https://mlsanity.com/paper/2309.02429/) (2023)
- [Cost-Aware Routing for Efficient Text-To-Image Generation](https://mlsanity.com/paper/2506.14753/) (2025)
- [Cost-Aware Evaluation and Model Scaling for LiDAR-Based 3D Object Detection](https://mlsanity.com/paper/2205.01142/) (2022)
- [Who Needs Decoders? Efficient Estimation of Sequence-level Attributes](https://mlsanity.com/paper/2305.05098/) (2023)
- [On Robustness to Missing Video for Audiovisual Speech Recognition](https://mlsanity.com/paper/2312.10088/) (2023)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [Training neural network ensembles via trajectory sampling](https://mlsanity.com/paper/2209.11116/) (2022)
- [Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification](https://mlsanity.com/paper/2310.16293/) (2023)
- [Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling](https://mlsanity.com/paper/2405.17139/) (2024)
- [Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering](https://mlsanity.com/paper/2512.02543/) (2025)
- [Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Models](https://mlsanity.com/paper/2308.12272/) (2023)
- [Regularized Neural Ensemblers](https://mlsanity.com/paper/2410.04520/) (2024)

## Abstract (arXiv)

Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.