mlsanityData to 7 Oct 2026

Smoothie: Label Free Language Model Routing

arXiv 2412.04692 ↗First posted 6 Dec 2024cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 65 of its 93 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

65 of 93 references.

References · 65

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  14. 2021True Few-Shot Learning with Language Modelscited by 343
  15. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  16. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  17. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  18. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  19. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  20. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  21. 2022Active Example Selection for In-Context Learningcited by 195
  22. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  23. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  24. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  25. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  26. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  27. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  28. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  29. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  30. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  31. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  32. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  33. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  34. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  35. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  36. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  37. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  38. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 12.

31 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025Learning to Orchestrate Agents in Natural Language with the Conductorcited by 22
  4. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  5. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  6. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  7. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  8. 2025An Adversary-Resistant Multi-Agent LLM System via Credibility Scoringcited by 10
  9. 2025LLMBoost: Make Large Language Models Stronger with Boostingcited by 8
  10. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedbackcited by 6
  11. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 5
  12. 2025TRINITY: An Evolved LLM Coordinatorcited by 5
19 more
  1. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
  2. 2025Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoningcited by 4
  3. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  4. 2026Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selectioncited by 4
  5. 2026Sakana Fugu Technical Reportcited by 4
  6. 2025MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verificationcited by 3
  7. 2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
  8. 2026Next Generation Active Learning: Mixture of LLMs in the Loopcited by 3
  9. 2025Routesplain: Towards Faithful and Intervenable Routing for Software-related Taskscited by 2
  10. 2026Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systemscited by 2
  11. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  12. 2026INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inferencecited by 1
  13. 2025CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervisioncited by 0
  14. 2025ECORE: Energy-Conscious Optimized Routing for Deep Learning Models at the Edgecited by 0
  15. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  16. 2026CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inferencecited by 0
  17. 2026Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modelingcited by 0
  18. 2026Learning to Fuse LLMs with Ontology Rankers for Rare-Disease Diagnosiscited by 0
  19. 2026MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichmentcited by 0