# Eagle: Efficient Training-Free Router for Multi-LLM Inference

arXiv: https://arxiv.org/abs/2409.15518 · HTML: https://mlsanity.com/paper/2409.15518/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM](https://mlsanity.com/paper/2401.02994/) (2024)
- [Plato: Plan to Efficiently Decode for Large Language Model Inference](https://mlsanity.com/paper/2402.12280/) (2024)

## Built on by (most cited first, 30 of 35)

- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution](https://mlsanity.com/paper/2506.17670/) (2025)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers](https://mlsanity.com/paper/2505.12601/) (2025)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Arch-Router: Aligning LLM Routing with Human Preferences](https://mlsanity.com/paper/2506.16655/) (2025)
- [C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning](https://mlsanity.com/paper/2511.07396/) (2025)
- [Bench-CoE: a Framework for Collaboration of Experts from Benchmark](https://mlsanity.com/paper/2412.04167/) (2024)
- [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://mlsanity.com/paper/2508.00234/) (2025)
- [LLM Routing with Dueling Feedback](https://mlsanity.com/paper/2510.00841/) (2025)
- [ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outliers](https://mlsanity.com/paper/2510.09852/) (2025)
- [RoboRouter: Training-Free Policy Routing for Robotic Manipulation](https://mlsanity.com/paper/2603.07892/) (2026)
- [Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization](https://mlsanity.com/paper/2604.15022/) (2026)
- [Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents](https://mlsanity.com/paper/2605.24598/) (2026)
- [One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection](https://mlsanity.com/paper/2509.09782/) (2025)
- [LLMRank: Understanding LLM Strengths for Model Routing](https://mlsanity.com/paper/2510.01234/) (2025)
- [Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems](https://mlsanity.com/paper/2602.11877/) (2026)
- [From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing](https://mlsanity.com/paper/2606.06924/) (2026)
- [Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents](https://mlsanity.com/paper/2502.04492/) (2025)
- [Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning](https://mlsanity.com/paper/2604.23623/) (2026)
- [Continual Model Routing in Evolving Model Hubs](https://mlsanity.com/paper/2605.28577/) (2026)
- [The Routing Plateau: Understanding the Accuracy Limits of LLM Routers](https://mlsanity.com/paper/2606.07587/) (2026)
- [RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing](https://mlsanity.com/paper/2606.18774/) (2026)
- [Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes](https://mlsanity.com/paper/2608.02415/) (2026)
- [LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving](https://mlsanity.com/paper/2608.08382/) (2026)
- [EMAFusion: A Self-Optimizing System for Seamless LLM Selection and Integration](https://mlsanity.com/paper/2504.10681/) (2025)
- [CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering](https://mlsanity.com/paper/2507.22937/) (2025)
- [PRISM-Consult: A Panel-of-Experts Architecture for Clinician-Aligned Diagnosis](https://mlsanity.com/paper/2510.01114/) (2025)

## Abstract (arXiv)

The proliferation of Large Language Models (LLMs) with varying capabilities and costs has created a need for efficient model selection in AI systems. LLM routers address this need by dynamically choosing the most suitable model for a given query based on task requirements and budget constraints. However, existing routers face challenges in scalability and real-time adaptation, particularly in high-volume online environments. We present Eagle, a novel LLM routing approach that combines global and local ELO ranking modules to overcome these limitations. By evaluating both general and specialized LLM abilities, Eagle provides a scalable, training-free solution that enhances model selection quality while reducing computational overhead. Our experiments across multiple datasets show Eagle consistently outperforms baseline methods, with improvements of up to 23.52 percent in Area Under Curve (AUC) scores. Moreover, Eagle demonstrates remarkable efficiency, requiring only 1/20 of baseline methods' time for initialization and 100 to 200 times faster incremental updates in online scenarios, making it well-suited for dynamic, high-volume online serving environments.