Guarded Query Routing for Large Language Models
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench, released as Python package gqr), covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (<4ms). The embedding-based fastText excels at speed (<1ms) with acceptable accuracy (80%), whereas LLMs yield the highest accuracy (91%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. Source code is available: https://github.com/williambrach/gqr.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 36 of its 51 references that are in the map. First posted on arXiv 20 May 2025.
Builds on
36 of 51 references.
References · 36
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2020GLU Variants Improve Transformercited by 1,804
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2016Enriching Word Vectors with Subword Informationcited by 1,583
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2016Bag of Tricks for Efficient Text Classificationcited by 876
24 more
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2017Self-Normalizing Neural Networkscited by 598
- 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 271
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
- 2023Large Language Model Routing with Benchmark Datasetscited by 172
- 2017DOC: Deep Open Classification of Text Documentscited by 109
- 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
- 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
- 2025Transformers without Normalizationcited by 70
- 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
- 2019Offensive Language and Hate Speech Detection for Danishcited by 49
- 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
- 2021Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLPcited by 17
- 2018An Improved Text Sentiment Classification Model Using TF-IDF and Next Word Negationcited by 16
- 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
- 2021Lifelong Learning on Evolving Graphs Under the Constraints of Imbalanced Classes and New Classescited by 11
- 2025Rerouting LLM Routerscited by 11
- 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
- 2025NANOGPT: A Query-Driven Large Language Model Retrieval-Augmented Generation System for Nanotechnology Researchcited by 6
- 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
- 2019Classifying textual data: shallow, deep and ensemble methodscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Lookahead Routing for Large Language Models
- 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
- 2025How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities
- 2025Adaptive LLM Routing under Budget Constraints
- 2026Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions
- 2026Rethinking Quality in Large Language Model Routing: Security-Conditioned Model-and-Budget Selection
- 2026Lightweight Query Routing for Adaptive RAG: A Baseline Study on RAGRouter-Bench
- 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing