# Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models

arXiv: https://arxiv.org/abs/2308.11601 · HTML: https://mlsanity.com/paper/2308.11601/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)

## Built on by (most cited first, 30 of 30)

- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [Meta Reasoning for Large Language Models](https://mlsanity.com/paper/2406.11698/) (2024)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2604.17148/) (2026)
- [Cascade-Aware Training of Language Models](https://mlsanity.com/paper/2406.00060/) (2024)
- [Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving](https://mlsanity.com/paper/2509.02718/) (2025)
- [Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers](https://mlsanity.com/paper/2505.12601/) (2025)
- [Rational Tuning of LLM Cascades via Probabilistic Modeling](https://mlsanity.com/paper/2501.09345/) (2025)
- [Towards Robust Multi-Modal Reasoning via Model Selection](https://mlsanity.com/paper/2310.08446/) (2023)
- [Efficiently Deploying LLMs with Controlled Risk](https://mlsanity.com/paper/2410.02173/) (2024)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [Economic Evaluation of LLMs](https://mlsanity.com/paper/2507.03834/) (2025)
- [CARGO: A Framework for Confidence-Aware Routing of Large Language Models](https://mlsanity.com/paper/2509.14899/) (2025)
- [Optimizing the Cost-Quality Tradeoff of Agentic Theorem Provers in Lean](https://mlsanity.com/paper/2606.04883/) (2026)
- [Herd: Using multiple, smaller LLMs to match the performances of proprietary, large LLMs via an intelligent composer](https://mlsanity.com/paper/2310.19902/) (2023)
- [CONCUR: A Framework for Continual Constrained and Unconstrained Routing](https://mlsanity.com/paper/2512.09386/) (2025)
- [Online Pandora's Box for Contextual LLM Cascading](https://mlsanity.com/paper/2606.07392/) (2026)
- [Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes](https://mlsanity.com/paper/2608.02415/) (2026)
- [Leveraging Open-Source Large Language Models for encoding Social Determinants of Health using an Intelligent Router](https://mlsanity.com/paper/2405.19631/) (2024)
- [A Greedy PDE Router for Blending Neural Operators and Classical Methods](https://mlsanity.com/paper/2509.24814/) (2025)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)
- [When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attack](https://mlsanity.com/paper/2605.17288/) (2026)
- [Codifying the Judge: Scalable Evaluation via Program Distillation](https://mlsanity.com/paper/2607.22561/) (2026)
- [Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers](https://mlsanity.com/paper/2610.02516/) (2026)

## Abstract (arXiv)

The introduction of the transformer architecture and the self-attention mechanism has led to an explosive production of language models trained on specific downstream tasks and data domains. With over 200, 000 models in the Hugging Face ecosystem, users grapple with selecting and optimizing models to suit multifaceted workflows and data domains while addressing computational, security, and recency concerns. There is an urgent need for machine learning frameworks that can eliminate the burden of model selection and customization and unleash the incredible power of the vast emerging model library for end users. Here, we propose a context-aware routing system, Tryage, that leverages a language model router for optimal selection of expert models from a model library based on analysis of individual input prompts. Inspired by the thalamic router in the brain, Tryage employs a perceptive router to predict down-stream model performance on prompts and, then, makes a routing decision using an objective function that integrates performance predictions with user goals and constraints that are incorporated through flags (e.g., model size, model recency). Tryage allows users to explore a Pareto front and automatically trade-off between task accuracy and secondary goals including minimization of model size, recency, security, verbosity, and readability. Across heterogeneous data sets that include code, text, clinical data, and patents, the Tryage framework surpasses Gorilla and GPT3.5 turbo in dynamic model selection identifying the optimal model with an accuracy of 50.9% , compared to 23.6% by GPT 3.5 Turbo and 10.8% by Gorilla. Conceptually, Tryage demonstrates how routing models can be applied to program and control the behavior of multi-model LLM systems to maximize efficient use of the expanding and evolving language model ecosystem.