Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally. Since models vary widely in capability and price, routing queries to models that balance quality and inference cost is essential. Existing router approaches assume access to centralized query-model evaluation data. However, these data are often fragmented across clients, such as end users and organizations, and are privacy-sensitive, which makes centralizing data infeasible. Additionally, per-client router training is ineffective since local evaluation data is limited and covers only a restricted query distribution and a biased subset of model evaluations. We introduce the first federated framework for LLM routing, enabling clients to learn a shared routing policy from local offline query-model evaluation data. Our framework supports both parametric multilayer perceptron router and nonparametric K-means router under heterogeneous client query distributions and non-uniform model coverage. Across three benchmarks, federated collaboration improves the accuracy-cost frontier over client-local routers, both via increased effective model coverage and better query generalization. Our theoretical results also validate that federated training reduces routing suboptimality.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
54 of 76 references.
References · 54
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
38 more
- 2019Towards Federated Learning at Scale: System Designcited by 832
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
- 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
- 2019Asynchronous Federated Optimizationcited by 267
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
- 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
- 2023AutoMix: Automatically Mixing Language Modelscited by 125
- 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
- 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
- 2021Heterogeneity for the Win: One-Shot Federated Clusteringcited by 87
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
- 2024A Unified Approach to Routing and Cascading for LLMscited by 80
- 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 72
- 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
- 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
- 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
- 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
- 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 48
- 2024Statistical optimal transportcited by 47
- 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
- 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
- 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routingcited by 34
- 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
- 2023Integration of Large Language Models and Federated Learningcited by 31
- 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 28
- 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
- 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 19
- 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
- 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
- 2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guaranteescited by 9
- 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
- 2025TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Taskscited by 9
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.