# Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

arXiv: https://arxiv.org/abs/2410.05434 · HTML: https://mlsanity.com/paper/2410.05434/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 47 of its 95 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 47)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)

## Built on by (most cited first, 24 of 24)

- [Process Reward Models for LLM Agents: Practical Framework and Directions](https://mlsanity.com/paper/2502.10325/) (2025)
- [All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning](https://mlsanity.com/paper/2503.01067/) (2025)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [Multi-Turn Code Generation Through Single-Step Rewards](https://mlsanity.com/paper/2502.20380/) (2025)
- [Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization](https://mlsanity.com/paper/2602.23008/) (2026)
- [Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs](https://mlsanity.com/paper/2510.00507/) (2025)
- [PABU: Progress-Aware Belief Update for Efficient LLM Agents](https://mlsanity.com/paper/2602.09138/) (2026)
- [Improving Interactive In-Context Learning from Natural Language Feedback](https://mlsanity.com/paper/2602.16066/) (2026)
- [Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale](https://mlsanity.com/paper/2511.08475/) (2025)
- [Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models](https://mlsanity.com/paper/2603.13985/) (2026)
- [A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training](https://mlsanity.com/paper/2607.04574/) (2026)
- [Distilling Realizable Students from Unrealizable Teachers](https://mlsanity.com/paper/2505.09546/) (2025)
- [Speculative Rollback Correction for Quality-Diverse Web Agent Imitation](https://mlsanity.com/paper/2606.12485/) (2026)
- [EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.21946/) (2026)
- [Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents](https://mlsanity.com/paper/2606.05558/) (2026)
- [What Drives Interactive Improvement from Feedback?](https://mlsanity.com/paper/2606.30774/) (2026)
- [CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedback](https://mlsanity.com/paper/2606.01830/) (2026)
- [When and What to Teach: Budget-Aware Online Adaptation for Web Agents](https://mlsanity.com/paper/2609.05513/) (2026)
- [The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks](https://mlsanity.com/paper/2609.25804/) (2026)
- [CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?](https://mlsanity.com/paper/2609.37216/) (2026)
- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)
- [SynAct: Synergistic Active Feedback Learning for LLM Agents](https://mlsanity.com/paper/2610.12651/) (2026)
- [Beyond Corrected Actions: Training LLM Agents on Textual Critiques](https://mlsanity.com/paper/2610.19367/) (2026)
- [Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents](https://mlsanity.com/paper/2610.19834/) (2026)

## Abstract (arXiv)

While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state -- information that is available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time. We evaluate LEAP on diverse decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves. We also provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with the student's realizability, which we empirically validate. Our code is available at https://leap-llm.github.io