mlsanityData to 14 Oct 2026

When to Reason: Semantic Router for vLLM

arXiv 2510.08731 ↗First posted 9 Oct 2025cs.ET cs.AI cs.CLcited by 12

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 13 of its 21 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

13 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis25 Aug
  2. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMs18 Aug
  3. 2025Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Models15 Aug

Other references · 10

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  7. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  9. 2024Semantic Routing for Enhanced Performance of LLM-Assisted Intent-Based 5G Core Network Management and Orchestrationcited by 8
  10. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  2. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 14
  3. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  4. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  5. 2026Is Escalation Worth It? On the Depth of LLM Cascadescited by 4
  6. 2026DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolutioncited by 1
  7. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  8. 2026Privacy-Preserving LLMs Routingcited by 1
  9. 2026Neural Router: Semantic Content Matching for Agentic AIcited by 1
  10. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  11. 2026Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Servingcited by 0
  12. 2026The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commercecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router
  2. 2024Performance Characterization of Expert Router for Scalable LLM Inference
  3. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
  4. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
  5. 2025TravelBench : Exploring LLM Performance in Low-Resource Domains
  6. 2025Lookahead Routing for Large Language Models
  7. 2026A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms
  8. 2025An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint