mlsanityData to 14 Oct 2026

IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs

arXiv 2509.06274 ↗First posted 8 Sep 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Routing incoming queries to the most cost-effective LLM while maintaining response quality poses a fundamental challenge in optimizing performance-cost trade-offs for large-scale commercial systems. We present IPR, -- ,a quality-constrained Intelligent Prompt Routing framework that dynamically selects optimal models based on predicted response quality and user-specified tolerance levels. IPR introduces three key innovations: (1) a modular architecture with lightweight quality estimators trained on 1.5M prompts annotated with calibrated quality scores, enabling fine-grained quality prediction across model families; (2) a user-controlled routing mechanism with tolerance parameter τ ∈ [0,1] that provides explicit control over quality-cost trade-offs; and (3) an extensible design using frozen encoders with model-specific adapters, reducing new model integration from days to hours. To rigorously train and evaluate IPR, we curate an industrial-level dataset IPRBenchfootnoteIPRBench will be released upon legal approval., a comprehensive benchmark containing 1.5 million examples with response quality annotations across 11 LLM candidates. Deployed on a major cloud platform, IPR achieves 43.9% cost reduction while maintaining quality parity with the strongest model in the Claude family and processes requests with sub-150ms latency. The deployed system and additional product details are publicly available at https://aws.amazon.com/bedrock/intelligent-prompt-routing/

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

30 of 41 references.

References · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  12. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
18 more
  1. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  3. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  4. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  5. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  6. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  7. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  8. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  9. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  10. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  11. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  12. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
  13. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  14. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  15. 2025CP-Router: An Uncertainty-Aware Router Between LLM and LRMcited by 20
  16. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routingcited by 13
  17. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  18. 2021AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarizationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  2. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  3. 2026Agentic Routing: The Harness-Native Data Flywheelcited by 4
  4. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  5. 2026Scalable Prompt Routing via Fine-Grained Latent Task Discoverycited by 1