# LLM-AutoDiff: Auto-Differentiate Any LLM Workflow

arXiv: https://arxiv.org/abs/2501.16673 · HTML: https://mlsanity.com/paper/2501.16673/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2025.

## Built on by (most cited first, 21 of 21)

- [Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management](https://mlsanity.com/paper/2510.06727/) (2025)
- [Defending Against Prompt Injection With a Few DefensiveTokens](https://mlsanity.com/paper/2507.07974/) (2025)
- [ReCreate: Reasoning and Creating Domain Agents Driven by Experience](https://mlsanity.com/paper/2601.11100/) (2026)
- [VeRO: A Harness for Agents to Optimize Agents](https://mlsanity.com/paper/2602.22480/) (2026)
- [HarnessOpt-Bench: Evaluating LLMs at Harness Optimization](https://mlsanity.com/paper/2608.06301/) (2026)
- [Scaling Textual Gradients via Sampling-Based Momentum](https://mlsanity.com/paper/2506.00400/) (2025)
- [MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games](https://mlsanity.com/paper/2603.09022/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement](https://mlsanity.com/paper/2606.27226/) (2026)
- [JudgeFlow: Agentic Workflow Optimization via Block Judge](https://mlsanity.com/paper/2601.07477/) (2026)
- [Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration](https://mlsanity.com/paper/2604.18131/) (2026)
- [VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents](https://mlsanity.com/paper/2606.05395/) (2026)
- [QUIVER: A Formal Framework for Quantifying Perturbation Propagation and Bifurcation in Compound AI Systems](https://mlsanity.com/paper/2605.23956/) (2026)
- [MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization](https://mlsanity.com/paper/2605.18869/) (2026)
- [HPFA: Hypergraph-Based Paired Failure Attribution for LLM Reasoning](https://mlsanity.com/paper/2608.02026/) (2026)
- [The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows](https://mlsanity.com/paper/2608.06714/) (2026)
- [LinGO: A Linguistic Graph Optimization Framework with LLMs for Interpreting Intents of Online Uncivil Discourse](https://mlsanity.com/paper/2602.04693/) (2026)
- [Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models](https://mlsanity.com/paper/2604.08335/) (2026)
- [FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients](https://mlsanity.com/paper/2604.26258/) (2026)
- [Segment-Level Agentic Topic Modeling for Improved Data Exploration and Resource Efficiency](https://mlsanity.com/paper/2609.31460/) (2026)
- [Joint Workflow and Prompt Optimization for User Behavior Simulation](https://mlsanity.com/paper/2610.07663/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have reshaped natural language processing, powering applications from multi-hop retrieval and question answering to autonomous agent workflows. Yet, prompt engineering -- the task of crafting textual inputs to effectively direct LLMs -- remains difficult and labor-intensive, particularly for complex pipelines that combine multiple LLM calls with functional operations like retrieval and data formatting. We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE) that extends textual gradient-based methods (such as Text-Grad) to multi-component, potentially cyclic LLM architectures. Implemented within the AdalFlow library, LLM-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine LLM to generate feedback-akin to textual gradients -- that guide iterative prompt updates. Unlike prior single-node approaches, LLM-AutoDiff inherently accommodates functional nodes, preserves time-sequential behavior in repeated calls (e.g., multi-hop loops), and combats the "lost-in-the-middle" problem by isolating distinct sub-prompts (instructions, formats, or few-shot examples). It further boosts training efficiency by focusing on error-prone samples through selective gradient computation. Across diverse tasks, including single-step classification, multi-hop retrieval-based QA, and agent-driven pipelines, LLM-AutoDiff consistently outperforms existing textual gradient baselines in both accuracy and training cost. By unifying prompt optimization through a graph-centric lens, LLM-AutoDiff offers a powerful new paradigm for scaling and automating LLM workflows - mirroring the transformative role that automatic differentiation libraries have long played in neural network research.