# IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

arXiv: https://arxiv.org/abs/2506.01048 · HTML: https://mlsanity.com/paper/2506.01048/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Neural Cognitive Diagnosis for Intelligent Education Systems](https://mlsanity.com/paper/1908.08733/) (2019)
- [Cost-Effective Online Multi-LLM Selection with Versatile Reward Models](https://mlsanity.com/paper/2405.16587/) (2024)
- [Routoo: Learning to Route to Large Language Models Effectively](https://mlsanity.com/paper/2401.13979/) (2024)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation](https://mlsanity.com/paper/2405.13622/) (2024)
- [Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models](https://mlsanity.com/paper/2409.03155/) (2024)
- [Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA](https://mlsanity.com/paper/2410.06524/) (2024)

## Built on by (most cited first, 30 of 40)

- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning](https://mlsanity.com/paper/2510.08439/) (2025)
- [R2-Router: A New Paradigm for LLM Routing with Reasoning](https://mlsanity.com/paper/2602.02823/) (2026)
- [LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations](https://mlsanity.com/paper/2602.09924/) (2026)
- [MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing](https://mlsanity.com/paper/2601.17814/) (2026)
- [Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs](https://mlsanity.com/paper/2510.17472/) (2025)
- [SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks](https://mlsanity.com/paper/2607.00053/) (2026)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [VL-RouterBench: A Benchmark for Vision-Language Model Routing](https://mlsanity.com/paper/2512.23562/) (2025)
- [RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs](https://mlsanity.com/paper/2509.25426/) (2025)
- [ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outliers](https://mlsanity.com/paper/2510.09852/) (2025)
- [LLM Router: Rethinking Routing with Prefill Activations](https://mlsanity.com/paper/2603.20895/) (2026)
- [JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory](https://mlsanity.com/paper/2509.22888/) (2025)
- [Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs](https://mlsanity.com/paper/2603.22206/) (2026)
- [Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations](https://mlsanity.com/paper/2601.22318/) (2026)
- [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026)
- [Routesplain: Towards Faithful and Intervenable Routing for Software-related Tasks](https://mlsanity.com/paper/2511.09373/) (2025)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)
- [Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information](https://mlsanity.com/paper/2608.23867/) (2026)
- [Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks](https://mlsanity.com/paper/2509.24186/) (2025)
- [The Routing Plateau: Understanding the Accuracy Limits of LLM Routers](https://mlsanity.com/paper/2606.07587/) (2026)
- [RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing](https://mlsanity.com/paper/2606.18774/) (2026)
- [Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools](https://mlsanity.com/paper/2607.18253/) (2026)
- [Error-Aware Reverse Auction Mechanism for Large Language Model Routing](https://mlsanity.com/paper/2608.12719/) (2026)
- [RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents](https://mlsanity.com/paper/2609.34712/) (2026)
- [Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration](https://mlsanity.com/paper/2602.04291/) (2026)
- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated exceptional performance across a wide range of natural language tasks. However, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. While powerful models deliver better results, they come at a high cost, whereas smaller models are more cost-effective but less capable. To address this trade-off, we propose IRT-Router, a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. Inspired by Item Response Theory (IRT), a psychological measurement methodology, IRT-Router explicitly models the relationship between LLM capabilities and user query attributes. This not only enables accurate prediction of response performance but also provides interpretable insights, such as LLM abilities and query difficulty. Additionally, we design an online query warm-up technique based on semantic similarity, further enhancing the online generalization capability of IRT-Router. Extensive experiments on 20 LLMs and 12 datasets demonstrate that IRT-Router outperforms most baseline methods in terms of effectiveness and interpretability. Its superior performance in cold-start scenarios further confirms the reliability and practicality of IRT-Router in real-world applications. Code is available at https://github.com/Mercidaiha/IRT-Router.