mlsanityData to 14 Oct 2026

You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

arXiv 2609.37902 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing LLM routers choose among models using static per-model costs. We show that open-weight inference markets introduce a second, largely ignored decision axis: after choosing a model, a client must still choose which provider serves it. Measuring live endpoints across [nummodels] open models, competing providers, multiple task types, and three measurement waves, we find that provider choice cannot be inferred from the price list. The same model can vary sharply in quality, latency, availability, and price across providers; higher-priced providers are consistently faster, but price does not reliably predict quality or availability; and provider feasibility is task-selective, with one deployment nearly normal on knowledge tasks but catastrophically degraded on multi-step reasoning. We formulate same-model provider selection as a price-taker market-aware routing problem. A simple measured-map policy routes to the cheapest provider that is both quality-equivalent and healthy, yielding matched-quality savings while avoiding degraded endpoints. Because the map drifts, we introduce FACET, an online provider router that certifies per-(provider x task) feasibility facets and fails safe to an anchor before serving uncertified arms. Across relaxed deployment assumptions, FACET tolerates imperfect task assignment and sparse feedback, while systematic evaluator bias exposes a quality-signal trust boundary that can be mitigated with ground-truth probes or audits. Live provider runs further confirm that certification can move real traffic from a premium anchor to a substantially cheaper certified endpoint. Our results suggest that market-aware LLM routing must measure not only which model to use, but also who serves it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

25 of 37 references.

References · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  12. 2013Bandits with Knapsackscited by 164
13 more
  1. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  2. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  3. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  4. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  5. 2024Evaluating Quantized Large Language Modelscited by 77
  6. 2020Stochastic Bandits with Linear Constraintscited by 68
  7. 2016Conservative Banditscited by 59
  8. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  9. 2018Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Banditcited by 50
  10. 2017A Change-Detection based Framework for Piecewise-stationary Multi-Armed Bandit Problemcited by 49
  11. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  12. 2026MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddingscited by 10
  13. 2026Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guaranteescited by 3

Built on by

Not cited yet.