mlsanityData to 14 Oct 2026

Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling

arXiv 2308.06077 ↗First posted 11 Aug 2023cs.CLcited by 95

Not on the map. Why?

Abstract (arXiv)

Generative language models (LMs) have become omnipresent across data science. For a wide variety of tasks, inputs can be phrased as natural language prompts for an LM, from whose output the solution can then be extracted. LM performance has consistently been increasing with model size - but so has the monetary cost of querying the ever larger models. Importantly, however, not all inputs are equally hard: some require larger LMs for obtaining a satisfactory solution, whereas for others smaller LMs suffice. Based on this fact, we design a framework for cost-effective language model choice, called "Fly-swat or cannon" (FORC). Given a set of inputs and a set of candidate LMs, FORC judiciously assigns each input to an LM predicted to do well on the input according to a so-called meta-model, aiming to achieve high overall performance at low cost. The cost-performance tradeoff can be flexibly tuned by the user. Options include, among others, maximizing total expected performance (or the number of processed inputs) while staying within a given cost budget, or minimizing total cost while processing all inputs. We evaluate FORC on 14 datasets covering five natural language tasks, using four candidate LMs of vastly different size and cost. With FORC, we match the performance of the largest available LM while achieving a cost reduction of 63%. Via our publicly available library, researchers as well as practitioners can thus save large amounts of money without sacrificing performance.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

17 of 27 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
5 more
  1. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  2. 2023How is ChatGPT's behavior changing over time?cited by 270
  3. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  4. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 22
  5. 2023Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Taskscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 22; 2025: 36; 2026: 32.

95 papers.

Sort:

By area:

  1. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  2. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  3. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  4. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  5. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  6. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 88
  7. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  8. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  9. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  10. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 72
  11. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  12. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
38 more
  1. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  2. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  3. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  4. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  5. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  6. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 33
  7. 2024Smoothie: Label Free Language Model Routingcited by 31
  8. 2024Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systemscited by 25
  9. 2025Adaptive LLM Routing under Budget Constraintscited by 24
  10. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 23
  11. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  12. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 19
  13. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 17
  14. 2024OptLLM: Optimal Assignment of Queries to Large Language Modelscited by 16
  15. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  16. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  17. 2024Cascade-Aware Training of Language Modelscited by 15
  18. 2024Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Modelscited by 13
  19. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routingcited by 13
  20. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  21. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
  22. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
  23. 2025Cascaded Language Models for Cost-effective Human-AI Decision-Makingcited by 11
  24. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  25. 2024Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenariocited by 10
  26. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 10
  27. 2026Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarioscited by 10
  28. 2026Agent-as-a-Router: Agentic Model Routing for Coding Taskscited by 10
  29. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 9
  30. 2025TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Taskscited by 9
  31. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  32. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  33. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  34. 2025LLM Routing with Dueling Feedbackcited by 7
  35. 2023Stranger Danger! Cross-Community Interactions with Fringe Users Increase the Growth of Fringe Communities on Redditcited by 5
  36. 2024Efficiently Deploying LLMs with Controlled Riskcited by 5
  37. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  38. 2025IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offscited by 5

50 most cited of 95 All 95 (JSON)