# Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs

arXiv: https://arxiv.org/abs/2406.16218 · HTML: https://mlsanity.com/paper/2406.16218/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)
- [Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation](https://mlsanity.com/paper/2310.02304/) (2023)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Language Agents as Optimizable Graphs](https://mlsanity.com/paper/2402.16823/) (2024)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation](https://mlsanity.com/paper/2403.01131/) (2024)
- [Are Large Language Models Good Prompt Optimizers?](https://mlsanity.com/paper/2402.02101/) (2024)
- [Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers](https://mlsanity.com/paper/2402.17564/) (2024)
- [LLF-Bench: Benchmark for Interactive Learning from Language Feedback](https://mlsanity.com/paper/2312.06853/) (2023)
- [Joint Prompt Optimization of Stacked LLMs using Variational Inference](https://mlsanity.com/paper/2306.12509/) (2023)
- [The Importance of Directional Feedback for LLM-based Optimizers](https://mlsanity.com/paper/2405.16434/) (2024)
- [Online Learning with Continuous Variations: Dynamic Regret and Reductions](https://mlsanity.com/paper/1902.07286/) (2019)

## Built on by (most cited first, 30 of 69)

- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies](https://mlsanity.com/paper/2502.02533/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Astra: A Multi-Agent System for GPU Kernel Performance Optimization](https://mlsanity.com/paper/2509.07506/) (2025)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play](https://mlsanity.com/paper/2411.00062/) (2024)
- [How to Correctly do Semantic Backpropagation on Language-based Agentic Systems](https://mlsanity.com/paper/2412.03624/) (2024)
- [Formalizing Learning from Language Feedback with Provable Guarantees](https://mlsanity.com/paper/2506.10341/) (2025)
- [DarwinX: Evolving Agent Harnesses Through Natural Selection](https://mlsanity.com/paper/2608.07545/) (2026)
- [Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Traces](https://mlsanity.com/paper/2605.10913/) (2026)
- [SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement](https://mlsanity.com/paper/2606.10546/) (2026)
- [Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces](https://mlsanity.com/paper/2410.15625/) (2024)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Recursive self-improvement of AI research agents](https://mlsanity.com/paper/2609.26457/) (2026)
- [HarnessOpt-Bench: Evaluating LLMs at Harness Optimization](https://mlsanity.com/paper/2608.06301/) (2026)
- [Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems](https://mlsanity.com/paper/2506.10060/) (2025)
- [Learning Game-Playing Agents with Generative Code Optimization](https://mlsanity.com/paper/2508.19506/) (2025)
- [Understanding the Challenges in Iterative Generative Optimization with LLMs](https://mlsanity.com/paper/2603.23994/) (2026)
- [Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning](https://mlsanity.com/paper/2502.08056/) (2025)
- [metaTextGrad: Automatically optimizing language model optimizers](https://mlsanity.com/paper/2505.18524/) (2025)
- [Learning, Fast and Slow: Towards LLMs That Adapt Continually](https://mlsanity.com/paper/2605.12484/) (2026)
- [MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems](https://mlsanity.com/paper/2605.28732/) (2026)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2411.14251/) (2024)
- [Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2506.08234/) (2025)
- [Fantastic Adaptive Taxonomies and How to Use Them](https://mlsanity.com/paper/2607.16387/) (2026)
- [Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses](https://mlsanity.com/paper/2609.05736/) (2026)
- [Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale](https://mlsanity.com/paper/2505.03973/) (2025)
- [P3: Prompts Promote Prompting](https://mlsanity.com/paper/2507.15675/) (2025)
- [A Two-Stage GPU Kernel Tuner Combining Semantic Refactoring and Search-Based Optimization](https://mlsanity.com/paper/2601.12698/) (2026)

## Abstract (arXiv)

We study a class of optimization problems motivated by automating the design and update of AI systems like coding assistants, robots, and copilots. AutoDiff frameworks, like PyTorch, enable efficient end-to-end optimization of differentiable systems. However, general computational workflows can be non-differentiable and involve rich feedback (e.g. console output or user's responses), heterogeneous parameters (e.g. prompts, codes), and intricate objectives (beyond maximizing a score). We investigate end-to-end generative optimization -- using generative models such as LLMs within the optimizer for automatic updating of general computational workflows. We discover that workflow execution traces are akin to back-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization. Formally, we frame a new mathematical setup, Optimization with Trace Oracle (OPTO). In OPTO, an optimizer receives an execution trace along with feedback on the computed output and updates parameters iteratively. We provide a Python library, Trace, that efficiently converts a workflow optimization problem into an OPTO instance using PyTorch-like syntax. Using Trace, we develop a general LLM-based generative optimizer called OptoPrime. In empirical studies, we find that OptoPrime is capable of first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, etc., and is often competitive with specialized optimizers for each domain. We envision Trace as an open research platform for devising novel generative optimizers and developing the next generation of interactive learning agents. Website: https://microsoft.github.io/Trace/.