# Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index

arXiv: https://arxiv.org/abs/2412.01690 · HTML: https://mlsanity.com/paper/2412.01690/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks](https://mlsanity.com/paper/2407.12994/) (2024)
- [Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies](https://mlsanity.com/paper/2406.06461/) (2024)
- [Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithm](https://mlsanity.com/paper/2408.11198/) (2024)
- [Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning](https://mlsanity.com/paper/2304.06962/) (2023)

## Built on by (most cited first, 4 of 4)

- [Cost-of-Pass: An Economic Framework for Evaluating Language Models](https://mlsanity.com/paper/2504.13359/) (2025)
- [BFS-PO: Best-First Search for Large Reasoning Models](https://mlsanity.com/paper/2602.14917/) (2026)
- [An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation](https://mlsanity.com/paper/2505.20182/) (2025)
- [Learning to Grade Efficiently: A Bandit-Driven Prompt-Selection Framework for Low-Cost LLM Essay Scoring](https://mlsanity.com/paper/2608.23814/) (2026)

## Abstract (arXiv)

As prompt engineering research rapidly evolves, evaluations beyond accuracy are crucial for developing cost-effective techniques. We present the Economical Prompting Index (EPI), a novel metric that combines accuracy scores with token consumption, adjusted by a user-specified cost concern level to reflect different resource constraints. Our study examines 6 advanced prompting techniques, including Chain-of-Thought, Self-Consistency, and Tree of Thoughts, across 10 widely-used language models and 4 diverse datasets. We demonstrate that approaches such as Self-Consistency often provide statistically insignificant gains while becoming cost-prohibitive. For example, on high-performing models like Claude 3.5 Sonnet, the EPI of simpler techniques like Chain-of-Thought (0.72) surpasses more complex methods like Self-Consistency (0.64) at slight cost concern levels. Our findings suggest a reevaluation of complex prompting strategies in resource-constrained scenarios, potentially reshaping future research priorities and improving cost-effectiveness for end-users.