LLMBridge: Reducing Costs to Access LLMs in a Prompt-Centric Internet
Not on the map. Why?
Abstract (arXiv)
Today's Internet infrastructure is centered around content retrieval over HTTP, with middleboxes (e.g., HTTP proxies) playing a crucial role in performance, security, and cost-effectiveness. We envision a future where Internet communication will be dominated by "prompts" sent to generative AI models. For this, we will need proxies that provide similar functions to HTTP proxies (e.g., caching, routing, compression) while dealing with unique challenges and opportunities of prompt-based communication. As a first step toward supporting prompt-based communication, we present LLMBridge, an LLM proxy designed for cost-conscious users, such as those in developing regions and education (e.g., students, instructors). LLMBridge supports three key optimizations: model selection (routing prompts to the most suitable model), context management (intelligently reducing the amount of context), and semantic caching (serving prompts using local models and vector databases). These optimizations introduce trade-offs between cost and quality, which applications navigate through a high-level, bidirectional interface. As case studies, we deploy LLMBridge in two cost-sensitive settings: a WhatsApp-based Q&A service and a university classroom environment. The WhatsApp service has been live for over twelve months, serving 100+ users and handling more than 14.7K requests. In parallel, we exposed LLMBridge to students across three computer science courses over a semester, where it supported diverse LLM-powered applications - such as reasoning agents and chatbots - and handled an average of 500 requests per day. We report on deployment experiences across both settings and use the collected workloads to benchmark the effectiveness of various cost-optimization strategies, analyzing their trade-offs in cost, latency, and response quality.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 63 references that are in the map. First posted on arXiv 4 Oct 2024.
Builds on
11 of 63 references.
References · 11
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019AutoML: A Survey of the State-of-the-Artcited by 390
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
- 2023Large Language Model Routing with Benchmark Datasetscited by 172
- 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
- 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
- 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reductioncited by 34
- 2024Towards Optimizing the Costs of LLM Usagecited by 33
- 2024Efficient Prompt Caching via Embedding Similaritycited by 9
- 2024TwIPS: A Large Language Model Powered Texting Application to Simplify Conversational Nuances for Autistic Userscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Rethinking Internet Communication Through LLMs: How Close Are We?
- 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
- 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
- 2024Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks
- 2024Reinforcement Learning for Optimizing RAG for Domain Chatbots
- 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- 2025PromptTailor: Multi-turn Intent-Aligned Prompt Synthesis for Lightweight LLMs
- 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities