FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelines
Not on the map. Why?
Abstract (arXiv)
Multi-step LLM pipelines fail through interactions among retrieval, reasoning, and formatting steps, so prompt-only optimization can miss bottlenecks in the chain. We present Fully Automated Prompt Optimization (FAPO), a framework that lets Claude Code optimize an LLM pipeline inside a standardized codebase. FAPO evaluates a pipeline, inspects intermediate steps, diagnoses failures, proposes scoped changes, and validates variants repeatedly to optimize against a score function. It first tries prompt edits and, only when prompt optimization appears insufficient, changes chain structure within the permitted scope when attribution identifies a structural bottleneck. Across six benchmarks and three task models, FAPO beats the baseline GEPA in 15 of 18 model-benchmark comparisons. In 11 model-benchmark comparisons, FAPO wins with non-overlapping mean ± trial-standard-deviation ranges, and the mean FAPO-GEPA gain is +14.1 pp. In the six HoVer and IFBench comparisons where prompt-first search escalated to structural changes, FAPO wins all six with a mean gain of +33.8 pp. FAPO also improves performance on security tasks: on CTIBench-RCM, a security CVE-to-CWE task, prompt-only FAPO lifts test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning. These results position FAPO as a state-of-the-art pipeline optimization technique for both general-purpose and security-focused tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
28 of 45 references.
References · 28
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2025Gemma 3 Technical Reportcited by 1,824
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2023Large Language Models as Optimizerscited by 864
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
- 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
- 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
16 more
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
- 2024TextGrad: Automatic "Differentiation" via Textcited by 273
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
- 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
- 2025Generalizing Verifiable Instruction Followingcited by 194
- 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
- 2024Best-of-N Jailbreakingcited by 76
- 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
- 2026AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimizationcited by 60
- 2026EvoX: Meta-Evolution for Automated Discoverycited by 60
- 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
- 2025Adversarial Reasoning at Jailbreaking Timecited by 29
- 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 23
- 2026Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMscited by 17
- 2026Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Reportcited by 7
- 2025Capability-Based Scaling Trends for LLM-Based Red-Teamingcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.