# The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models

arXiv: https://arxiv.org/abs/2308.08061 · HTML: https://mlsanity.com/paper/2308.08061/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 12 of 12)

- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [A cost-benefit analysis of cross-lingual transfer methods](https://mlsanity.com/paper/2105.06813/) (2021)
- [The Future of ChatGPT-enabled Labor Market: A Preliminary Study in China](https://mlsanity.com/paper/2304.09823/) (2023)

## Built on by (most cited first, 5 of 5)

- [Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management](https://mlsanity.com/paper/2505.03756/) (2025)
- [Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework](https://mlsanity.com/paper/2506.05695/) (2025)
- [JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensembles](https://mlsanity.com/paper/2410.08442/) (2024)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [Getting In Contract with Large Language Models -- An Agency Theory Perspective On Large Language Model Alignment](https://mlsanity.com/paper/2509.07642/) (2025)

## Abstract (arXiv)

When deploying machine learning models in production for any product/application, there are three properties that are commonly desired. First, the models should be generalizable, in that we can extend it to further use cases as our knowledge of the domain area develops. Second they should be evaluable, so that there are clear metrics for performance and the calculation of those metrics in production settings are feasible. Finally, the deployment should be cost-optimal as far as possible. In this paper we propose that these three objectives (i.e. generalization, evaluation and cost-optimality) can often be relatively orthogonal and that for large language models, despite their performance over conventional NLP models, enterprises need to carefully assess all the three factors before making substantial investments in this technology. We propose a framework for generalization, evaluation and cost-modeling specifically tailored to large language models, offering insights into the intricacies of development, deployment and management for these large language models.