# Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

arXiv: https://arxiv.org/abs/2401.02994 · HTML: https://mlsanity.com/paper/2401.02994/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://mlsanity.com/paper/2006.16779/) (2020)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Ensemble Distillation for Neural Machine Translation](https://mlsanity.com/paper/1702.01802/) (2017)
- [A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLP](https://mlsanity.com/paper/2104.10810/) (2021)
- [BARD: A structured technique for group elicitation of Bayesian networks to support analytic reasoning](https://mlsanity.com/paper/2003.01207/) (2020)
- [Ensemble Distillation Approaches for Grammatical Error Correction](https://mlsanity.com/paper/2012.07535/) (2020)
- [CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models](https://mlsanity.com/paper/2306.05317/) (2023)
- [Grounding in social media: An approach to building a chit-chat dialogue model](https://mlsanity.com/paper/2206.05696/) (2022)
- [Minimum Bayes' Risk Decoding for System Combination of Grammatical Error Correction Systems](https://mlsanity.com/paper/2309.06520/) (2023)

## Built on by (most cited first, 27 of 27)

- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [Eagle: Efficient Training-Free Router for Multi-LLM Inference](https://mlsanity.com/paper/2409.15518/) (2024)
- [Smoothie: Label Free Language Model Routing](https://mlsanity.com/paper/2412.04692/) (2024)
- [Meta Reasoning for Large Language Models](https://mlsanity.com/paper/2406.11698/) (2024)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [Adaptive LLM Routing under Budget Constraints](https://mlsanity.com/paper/2508.21141/) (2025)
- [SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts](https://mlsanity.com/paper/2405.07518/) (2024)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [LLM-based multi-agent poetry generation in non-cooperative environments](https://mlsanity.com/paper/2409.03659/) (2024)
- [Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers](https://mlsanity.com/paper/2505.12601/) (2025)
- [LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planning](https://mlsanity.com/paper/2409.01806/) (2024)
- [TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Tasks](https://mlsanity.com/paper/2506.12473/) (2025)
- [Post-Hoc Reversal: Are We Selecting Models Prematurely?](https://mlsanity.com/paper/2404.07815/) (2024)
- [Bridging Language Models and Financial Analysis](https://mlsanity.com/paper/2503.22693/) (2025)
- [Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process](https://mlsanity.com/paper/2512.23213/) (2025)
- [From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialogue](https://mlsanity.com/paper/2604.12385/) (2026)
- [Sapiens2](https://mlsanity.com/paper/2604.21681/) (2026)
- [Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuning](https://mlsanity.com/paper/2506.02519/) (2025)
- [S2GPT-PINNs: Sparse and Small models for PDEs](https://mlsanity.com/paper/2506.15687/) (2025)
- [Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models](https://mlsanity.com/paper/2512.15089/) (2025)
- [It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisation](https://mlsanity.com/paper/2503.02463/) (2025)
- [Teaming LLMs to Detect and Mitigate Hallucinations](https://mlsanity.com/paper/2510.19507/) (2025)
- [On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems](https://mlsanity.com/paper/2406.13720/) (2024)
- [Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs](https://mlsanity.com/paper/2610.05463/) (2026)

## Abstract (arXiv)

In conversational AI research, there's a noticeable trend towards developing models with a larger number of parameters, exemplified by models like ChatGPT. While these expansive models tend to generate increasingly better chat responses, they demand significant computational resources and memory. This study explores a pertinent question: Can a combination of smaller models collaboratively achieve comparable or enhanced performance relative to a singular large model? We introduce an approach termed "blending", a straightforward yet effective method of integrating multiple chat AIs. Our empirical evidence suggests that when specific smaller models are synergistically blended, they can potentially outperform or match the capabilities of much larger counterparts. For instance, integrating just three models of moderate size (6B/13B paramaeters) can rival or even surpass the performance metrics of a substantially larger model like ChatGPT (175B+ paramaters). This hypothesis is rigorously tested using A/B testing methodologies with a large user base on the Chai research platform over a span of thirty days. The findings underscore the potential of the "blending" strategy as a viable approach for enhancing chat AI efficacy without a corresponding surge in computational demands.