# Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference

arXiv: https://arxiv.org/abs/2303.04673 · HTML: https://mlsanity.com/paper/2303.04673/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation](https://mlsanity.com/paper/2009.07243/) (2020)
- [An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://mlsanity.com/paper/2106.09204/) (2021)
- [Leveraging Natural Supervision for Language Representation Learning and Generation](https://mlsanity.com/paper/2207.10617/) (2022)
- [Don't Be So Sure! Boosting ASR Decoding via Confidence Relaxation](https://mlsanity.com/paper/2212.13378/) (2022)

## Built on by (most cited first, 30 of 40)

- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024)
- [EcoAssistant: Using LLM Assistant More Affordably and Accurately](https://mlsanity.com/paper/2310.03046/) (2023)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Large Language Models as Test Case Generators: Performance Evaluation and Enhancement](https://mlsanity.com/paper/2404.13340/) (2024)
- [ReCode: Reinforcing Code Generation with Reasoning-Process Rewards](https://mlsanity.com/paper/2508.05170/) (2025)
- [FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch](https://mlsanity.com/paper/2406.04824/) (2024)
- [Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning](https://mlsanity.com/paper/2505.24478/) (2025)
- [Scaling LLM Inference with Optimized Sample Compute Allocation](https://mlsanity.com/paper/2410.22480/) (2024)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [CEval: A Benchmark for Evaluating Counterfactual Text Generation](https://mlsanity.com/paper/2404.17475/) (2024)
- [AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces](https://mlsanity.com/paper/2608.23041/) (2026)
- [AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.19251/) (2024)
- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems](https://mlsanity.com/paper/2502.18635/) (2025)
- [Problem Solving Through Human-AI Preference-Based Cooperation](https://mlsanity.com/paper/2408.07461/) (2024)
- [EcoAct: Economic Agent Determines When to Register What Action](https://mlsanity.com/paper/2411.01643/) (2024)
- [MS-BART: Unified Modeling of Mass Spectra and Molecules for Structure Elucidation](https://mlsanity.com/paper/2510.20615/) (2025)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [Scaling Textual Gradients via Sampling-Based Momentum](https://mlsanity.com/paper/2506.00400/) (2025)
- [The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines](https://mlsanity.com/paper/2408.01050/) (2024)
- [LLM can Achieve Self-Regulation via Hyperparameter Aware Generation](https://mlsanity.com/paper/2402.11251/) (2024)
- [Guiding Large Language Models to Generate Computer-Parsable Content](https://mlsanity.com/paper/2404.05499/) (2024)
- [Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement](https://mlsanity.com/paper/2409.02428/) (2024)
- [HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimization](https://mlsanity.com/paper/2502.01276/) (2025)
- [Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning](https://mlsanity.com/paper/2601.22323/) (2026)
- [Return of EM: Entity-driven Answer Set Expansion for QA Evaluation](https://mlsanity.com/paper/2404.15650/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have sparked significant interest in their generative capabilities, leading to the development of various commercial applications. The high cost of using the models drives application builders to maximize the value of generation under a limited inference budget. This paper presents a study of optimizing inference hyperparameters such as the number of responses, temperature and max tokens, which significantly affects the utility/cost of text generation. We design a framework named EcoOptiGen which leverages economical hyperparameter optimization and cost-based pruning. Experiments with the GPT-3.5/GPT-4 models on a variety of tasks verify its effectiveness. EcoOptiGen is implemented in the `autogen' package of the FLAML library: urlhttps://aka.ms/autogen.