# Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights

arXiv: https://arxiv.org/abs/2407.14788 · HTML: https://mlsanity.com/paper/2407.14788/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2024.

## Built on by (most cited first, 4 of 4)

- [Provable Scaling Laws for the Test-Time Compute of Large Language Models](https://mlsanity.com/paper/2411.19477/) (2024)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)
- [EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs](https://mlsanity.com/paper/2505.17139/) (2025)

## Abstract (arXiv)

This work presents an analytical framework for the design and analysis of LLM-based algorithms, i.e., algorithms that contain one or multiple calls of large language models (LLMs) as sub-routines and critically rely on the capabilities of LLMs. While such algorithms, ranging from basic LLM calls with prompt engineering to complicated LLM-powered agentic workflows and compound AI systems, have achieved remarkable empirical success, their design and optimization oftentimes require extensive trial-and-errors and case-by-case analysis. Our proposed framework serves as an attempt to mitigate such headaches, offering a formal and systematic approach for analyzing how the accuracy and efficiency of an LLM-based algorithm will be impacted by critical design choices, such as the pattern and granularity of task decomposition, or the prompt for each LLM call. Through a wide range of case studies covering diverse algorithm patterns (including parallel/hierarchical/recursive task decomposition and generic directed acyclic graphs), we demonstrate the proposed framework in action and derive interesting insights that generalize across scenarios, accompanied by systematic empirical validation in synthetic settings.