mlsanityData to 7 Oct 2026

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

arXiv 2305.05176 ↗First posted 9 May 2023cs.LG cs.AI cs.CLcited by 791

Not on the map. Why?

Cited by 191 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

Key facts. Cited by 791 papers in the map (data through 7 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

33 of 57 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
21 more
  1. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  2. 2018CoQA: A Conversational Question Answering Challengecited by 674
  3. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  4. 2023Augmented Language Models: a Surveycited by 398
  5. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  6. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  7. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  8. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  9. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  10. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  11. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  12. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  13. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  14. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  15. 2016Weakly Supervised Cascaded Convolutional Networkscited by 47
  16. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  17. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  18. 2015Learning Complexity-Aware Cascades for Deep Pedestrian Detectioncited by 18
  19. 2021Did the Model Change? Efficiently Assessing Machine Learning API Shiftscited by 13
  20. 2021Efficient Online ML API Selection for Multi-Label Classification Taskscited by 11
  21. 2022Semi-Supervised Cascaded Clustering for Classification of Noisy Label Datacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 46; 2024: 87; 2025: 173; 2026: 485.

791 papers.

Sort:

By area:

  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  2. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  4. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  6. 2023Large Language Models as Tool Makerscited by 293
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2024More Agents Is All You Needcited by 174
  11. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  12. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
38 more
  1. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  2. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  3. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  4. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  5. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  7. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  8. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  9. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  10. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  11. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  12. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  13. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  14. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  15. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  16. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  17. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  18. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  19. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  20. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  21. 2023Large Language Models for Supply Chain Optimizationcited by 61
  22. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  23. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  24. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  25. 2023Lyfe Agents: Generative agents for low-cost real-time social interactionscited by 49
  26. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  27. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 47
  28. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  29. 2023Fusing Models with Complementary Expertisecited by 45
  30. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  31. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  32. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  33. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  34. 2023LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMscited by 40
  35. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  36. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  37. 2024Faster Cascades via Speculative Decodingcited by 38
  38. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37

50 most cited of 791 All 791 (JSON)