# Towards Optimizing the Costs of LLM Usage

arXiv: https://arxiv.org/abs/2402.01742 · HTML: https://mlsanity.com/paper/2402.01742/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 13 of 13)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning](https://mlsanity.com/paper/2010.15875/) (2020)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply](https://mlsanity.com/paper/2006.07512/) (2020)
- [Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts](https://mlsanity.com/paper/2011.04864/) (2020)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [TangoBERT: Reducing Inference Cost by using Cascaded Architecture](https://mlsanity.com/paper/2204.06271/) (2022)

## Built on by (most cited first, 30 of 33)

- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [KG-RAG: Bridging the Gap Between Knowledge and Creativity](https://mlsanity.com/paper/2405.12035/) (2024)
- [Optimizing Model Selection for Compound AI Systems](https://mlsanity.com/paper/2502.14815/) (2025)
- [Causal LLM Routing: End-to-End Regret Minimization from Observational Data](https://mlsanity.com/paper/2505.16037/) (2025)
- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation](https://mlsanity.com/paper/2409.07355/) (2024)
- [Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors](https://mlsanity.com/paper/2509.14543/) (2025)
- [SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving](https://mlsanity.com/paper/2501.03535/) (2025)
- [Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning](https://mlsanity.com/paper/2410.08014/) (2024)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)
- ["X of Information'' Continuum: A Survey on AI-Driven Multi-dimensional Metrics for Next-Generation Networked Systems](https://mlsanity.com/paper/2507.19657/) (2025)
- [RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification](https://mlsanity.com/paper/2505.18380/) (2025)
- [The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More](https://mlsanity.com/paper/2603.23971/) (2026)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [Engineering Trustworthy AI: A Developer Guide for Empirical Risk Minimization](https://mlsanity.com/paper/2410.19361/) (2024)
- [Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights](https://mlsanity.com/paper/2504.06307/) (2025)
- [LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text](https://mlsanity.com/paper/2505.24826/) (2025)
- [Dynamic System Instructions and Tool Exposure for Efficient Agentic LLMs](https://mlsanity.com/paper/2602.17046/) (2025)
- [FAA Framework: A Large Language Model-Based Approach for Credit Card Fraud Investigations](https://mlsanity.com/paper/2506.11635/) (2025)
- [eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing](https://mlsanity.com/paper/2508.10370/) (2025)
- [Effective and Efficient Threat Hunting with Small Language Models](https://mlsanity.com/paper/2512.06660/) (2025)
- [Multi-LLM Query Optimization](https://mlsanity.com/paper/2603.24617/) (2026)
- [Online Learning with LLM Experts from Limited Feedback](https://mlsanity.com/paper/2609.05820/) (2026)
- [LLMBridge: Reducing Costs to Access LLMs in a Prompt-Centric Internet](https://mlsanity.com/paper/2410.11857/) (2024)
- [Software Performance Engineering for Foundation Model-Powered Software](https://mlsanity.com/paper/2411.09580/) (2024)
- [Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization](https://mlsanity.com/paper/2505.15835/) (2025)
- [On-Device Large Language Models for Sequential Recommendation](https://mlsanity.com/paper/2601.09306/) (2026)
- [From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utility](https://mlsanity.com/paper/2602.20513/) (2026)
- [RAGent: Physics-Aware Agentic Reasoning for Training-Free mmWave Human Activity Recognition](https://mlsanity.com/paper/2603.27571/) (2026)
- [Dialogue to Discovery: Attribute-Aware Preference Elicitation for Conversational Product Search Assistants](https://mlsanity.com/paper/2606.24194/) (2026)

## Abstract (arXiv)

Generative AI and LLMs in particular are heavily used nowadays for various document processing tasks such as question answering and summarization. However, different LLMs come with different capabilities for different tasks as well as with different costs, tokenization, and latency. In fact, enterprises are already incurring huge costs of operating or using LLMs for their respective use cases. In this work, we propose optimizing the usage costs of LLMs by estimating their output quality (without actually invoking the LLMs), and then solving an optimization routine for the LLM selection to either keep costs under a budget, or minimize the costs, in a quality and latency aware manner. We propose a model to predict the output quality of LLMs on document processing tasks like summarization, followed by an LP rounding algorithm to optimize the selection of LLMs. We study optimization problems trading off the quality and costs, both theoretically and empirically. We further propose a sentence simplification model for reducing the number of tokens in a controlled manner. Additionally, we propose several deterministic heuristics for reducing tokens in a quality aware manner, and study the related optimization problem of applying the heuristics optimizing the quality and cost trade-off. We perform extensive empirical validation of our methods on not only enterprise datasets but also on open-source datasets, annotated by us, and show that we perform much better compared to closest baselines. Our methods reduce costs by 40%- 90% while improving quality by 4%-7%. We will release the annotated open source datasets to the community for further research and exploration.