# OptLLM: Optimal Assignment of Queries to Large Language Models

arXiv: https://arxiv.org/abs/2405.15130 · HTML: https://mlsanity.com/paper/2405.15130/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018)
- [Log Parsing: How Far Can ChatGPT Go?](https://mlsanity.com/paper/2306.01590/) (2023)
- [Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks](https://mlsanity.com/paper/2301.07006/) (2023)
- [Beyond Evolutionary Algorithms for Search-based Software Engineering](https://mlsanity.com/paper/1701.07950/) (2017)

## Built on by (most cited first, 16 of 16)

- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [LLM Routing with Dueling Feedback](https://mlsanity.com/paper/2510.00841/) (2025)
- [PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databases](https://mlsanity.com/paper/2506.17336/) (2025)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection](https://mlsanity.com/paper/2509.09782/) (2025)
- [CONCUR: A Framework for Continual Constrained and Unconstrained Routing](https://mlsanity.com/paper/2512.09386/) (2025)
- [ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models](https://mlsanity.com/paper/2603.21237/) (2026)
- [LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters](https://mlsanity.com/paper/2607.17175/) (2026)
- [Unsupervised Query Routing for Retrieval Augmented Generation](https://mlsanity.com/paper/2501.07793/) (2025)
- [KRONE: Scalable LLM-Augmented Log Anomaly Detection via Hierarchical Abstraction](https://mlsanity.com/paper/2602.07303/) (2026)
- [Expected Reward Prediction, with Applications to Model Routing](https://mlsanity.com/paper/2603.20217/) (2026)
- [RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning](https://mlsanity.com/paper/2608.15817/) (2026)
- [DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments](https://mlsanity.com/paper/2609.00442/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have garnered considerable attention owing to their remarkable capabilities, leading to an increasing number of companies offering LLMs as services. Different LLMs achieve different performance at different costs. A challenge for users lies in choosing the LLMs that best fit their needs, balancing cost and performance. In this paper, we propose a framework for addressing the cost-effective query allocation problem for LLMs. Given a set of input queries and candidate LLMs, our framework, named OptLLM, provides users with a range of optimal solutions to choose from, aligning with their budget constraints and performance preferences, including options for maximizing accuracy and minimizing cost. OptLLM predicts the performance of candidate LLMs on each query using a multi-label classification model with uncertainty estimation and then iteratively generates a set of non-dominated solutions by destructing and reconstructing the current solution. To evaluate the effectiveness of OptLLM, we conduct extensive experiments on various types of tasks, including text classification, question answering, sentiment analysis, reasoning, and log parsing. Our experimental results demonstrate that OptLLM substantially reduces costs by 2.40% to 49.18% while achieving the same accuracy as the best LLM. Compared to other multi-objective optimization algorithms, OptLLM improves accuracy by 2.94% to 69.05% at the same cost or saves costs by 8.79% and 95.87% while maintaining the highest attainable accuracy.