RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Not on the map. Why?
Abstract (arXiv)
We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/LAMDA-Model-Reuse/ORBIT.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
34 of 38 references.
References · 34
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
- 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
- 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
- 2023AutoMix: Automatically Mixing Language Modelscited by 125
- 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
22 more
- 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
- 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
- 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
- 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
- 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
- 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
- 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 48
- 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
- 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
- 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
- 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 35
- 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
- 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
- 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 33
- 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 28
- 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
- 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
- 2023When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTscited by 13
- 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
- 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
- 2024Beimingwu: A Learnware Dock Systemcited by 8
- 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routingcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
- 2026R2-Router: A New Paradigm for LLM Routing with Reasoning
- 2025How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities
- 2026Learning to Route LLMs from Implicit Cost-Performance Preferences via Meta-Learning
- 2025PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling
- 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
- 2026RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing
- 2025Dynamic LLM Routing and Selection based on User Preferences: Balancing Performance, Cost, and Ethics