# Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards

arXiv: https://arxiv.org/abs/2507.03041 · HTML: https://mlsanity.com/paper/2507.03041/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 30 of 42)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies](https://mlsanity.com/paper/2502.02533/) (2025)
- [On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents](https://mlsanity.com/paper/2408.00989/) (2024)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [Process Reward Models for LLM Agents: Practical Framework and Directions](https://mlsanity.com/paper/2502.10325/) (2025)
- [Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS](https://mlsanity.com/paper/1912.06059/) (2019)
- [Step-level Value Preference Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2406.10858/) (2024)
- [AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning](https://mlsanity.com/paper/2406.11200/) (2024)

## Built on by (most cited first, 24 of 24)

- [AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?](https://mlsanity.com/paper/2509.03312/) (2025)
- [Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms](https://mlsanity.com/paper/2508.18298/) (2025)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [Sherlock: Reliable and Efficient Agentic Workflow Execution](https://mlsanity.com/paper/2511.00330/) (2025)
- [When Does Continual Learning Require Learning](https://mlsanity.com/paper/2607.07847/) (2026)
- [Learning, Fast and Slow: Towards LLMs That Adapt Continually](https://mlsanity.com/paper/2605.12484/) (2026)
- [TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuning](https://mlsanity.com/paper/2602.08306/) (2026)
- [Causal methods for LLM development and evaluation](https://mlsanity.com/paper/2605.25998/) (2026)
- [IM-Chat: A Multi-agent LLM Framework Integrating Tool-Calling and Diffusion Modeling for Knowledge Transfer in Injection Molding Industry](https://mlsanity.com/paper/2507.15268/) (2025)
- [Textual Equilibrium Propagation for Deep Compound AI Systems](https://mlsanity.com/paper/2601.21064/) (2026)
- [ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs](https://mlsanity.com/paper/2602.09475/) (2026)
- [Joint Optimization of Multi-agent Memory System](https://mlsanity.com/paper/2603.12631/) (2026)
- [Compass: Optimizing Compound AI Workflows for Dynamic Adaptation](https://mlsanity.com/paper/2603.20821/) (2026)
- [A Sober Look at Agentic Misalignment in Automated Workflows](https://mlsanity.com/paper/2605.24197/) (2026)
- [Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems](https://mlsanity.com/paper/2606.14350/) (2026)
- [CAFE: A Compound-AI Factorial Evaluation Framework](https://mlsanity.com/paper/2607.10380/) (2026)
- [MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graph](https://mlsanity.com/paper/2608.10504/) (2026)
- [Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems](https://mlsanity.com/paper/2607.21627/) (2026)
- [ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize](https://mlsanity.com/paper/2609.04197/) (2026)
- [Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusters](https://mlsanity.com/paper/2609.04513/) (2026)
- [Communication between Frozen Large Language Models via Prompt Optimization in a Referential Game](https://mlsanity.com/paper/2609.31989/) (2026)
- [What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents](https://mlsanity.com/paper/2609.32318/) (2026)
- [It Takes Workflows to Evolve Better Workflows](https://mlsanity.com/paper/2610.01026/) (2026)
- [Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting](https://mlsanity.com/paper/2610.12601/) (2026)

## Abstract (arXiv)

Compound AI systems integrating multiple components, such as Large Language Models, specialized tools, and traditional machine learning models, are increasingly deployed to solve complex real-world tasks. However, optimizing compound systems remains challenging due to their non-differentiable structures and diverse configuration types across components, including prompts, hyperparameters, and model parameters. To address this challenge, we propose Optimas, a unified framework for effective optimization of compound systems. The core idea of Optimas is to maintain one Local Reward Function (LRF) per component, each satisfying a local-global alignment property, i.e., each component's local reward correlates with the global system performance. In each iteration, Optimas efficiently adapts the LRFs to maintain this property while simultaneously maximizing each component's local reward. This approach enables independent updates of heterogeneous configurations using the designated optimization method, while ensuring that local improvements consistently lead to performance gains. We present extensive evaluations across five real-world compound systems to demonstrate that Optimas outperforms strong baselines by an average improvement of 11.92%, offering a general and effective approach for improving compound systems. Our website is at https://optimas.stanford.edu.