# Smoothie: Label Free Language Model Routing

arXiv: https://arxiv.org/abs/2412.04692 · HTML: https://mlsanity.com/paper/2412.04692/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 93 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 65)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)

## Built on by (most cited first, 30 of 31)

- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Learning to Orchestrate Agents in Natural Language with the Conductor](https://mlsanity.com/paper/2512.04388/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR](https://mlsanity.com/paper/2504.11101/) (2025)
- [An Adversary-Resistant Multi-Agent LLM System via Credibility Scoring](https://mlsanity.com/paper/2505.24239/) (2025)
- [LLMBoost: Make Large Language Models Stronger with Boosting](https://mlsanity.com/paper/2512.22309/) (2025)
- [On Calibration of Large Language Models: From Response To Capability](https://mlsanity.com/paper/2602.13540/) (2026)
- [Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback](https://mlsanity.com/paper/2508.12338/) (2025)
- [Uncertainty Quantification for Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2510.11483/) (2025)
- [TRINITY: An Evolved LLM Coordinator](https://mlsanity.com/paper/2512.04695/) (2025)
- [Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2511.02130/) (2025)
- [Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process](https://mlsanity.com/paper/2512.23213/) (2025)
- [Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection](https://mlsanity.com/paper/2601.09692/) (2026)
- [Sakana Fugu Technical Report](https://mlsanity.com/paper/2606.21228/) (2026)
- [MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verification](https://mlsanity.com/paper/2508.02584/) (2025)
- [Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensembling](https://mlsanity.com/paper/2509.21091/) (2025)
- [Next Generation Active Learning: Mixture of LLMs in the Loop](https://mlsanity.com/paper/2601.15773/) (2026)
- [Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems](https://mlsanity.com/paper/2602.11877/) (2026)
- [Routesplain: Towards Faithful and Intervenable Routing for Software-related Tasks](https://mlsanity.com/paper/2511.09373/) (2025)
- [INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference](https://mlsanity.com/paper/2605.18853/) (2026)
- [Rubric-Guided Process Reward for Stepwise Model Routing](https://mlsanity.com/paper/2605.29310/) (2026)
- [CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference](https://mlsanity.com/paper/2605.12001/) (2026)
- [CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision](https://mlsanity.com/paper/2506.14912/) (2025)
- [ECORE: Energy-Conscious Optimized Routing for Deep Learning Models at the Edge](https://mlsanity.com/paper/2507.06011/) (2025)
- [PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Prompts](https://mlsanity.com/paper/2603.21438/) (2026)
- [Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling](https://mlsanity.com/paper/2608.15565/) (2026)
- [Learning to Fuse LLMs with Ontology Rankers for Rare-Disease Diagnosis](https://mlsanity.com/paper/2609.02473/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.