mlsanityData to 14 Oct 2026

Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

arXiv 2410.05434 ↗First posted 7 Oct 2024cs.LG cs.AIcited by 24

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state -- information that is available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time. We evaluate LEAP on diverse decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves. We also provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with the student's realizability, which we empirically validate. Our code is available at https://leap-llm.github.io

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 47 of its 95 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

47 of 95 references.

References · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
35 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  4. 2023Teaching Large Language Models to Self-Debugcited by 988
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  8. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  9. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  10. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  11. 2019Learning by Cheatingcited by 383
  12. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  13. 2022Generating Sequences by Learning to Self-Correctcited by 277
  14. 2024Iterative Reasoning Preference Optimizationcited by 246
  15. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  16. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  17. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  18. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  20. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  21. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  22. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  23. 2023Shepherd: A Critic for Language Model Generationcited by 97
  24. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 93
  25. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 71
  26. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  27. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 62
  28. 2015Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Searchcited by 42
  29. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  30. 2017Data-driven Planning via Imitation Learningcited by 36
  31. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  32. 2024SPIN: Simultaneous Perception, Interaction and Navigationcited by 29
  33. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 19
  34. 2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 18
  35. 2022Fighting Fire with Fire: Avoiding DNN Shortcuts through Primingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 17.

24 papers.

Sort:

By area:

  1. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
  2. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  3. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 29
  4. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 28
  5. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  6. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  7. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 7
  8. 2026Improving Interactive In-Context Learning from Natural Language Feedbackcited by 5
  9. 2025Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationalecited by 3
  10. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  11. 2026A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Trainingcited by 3
  12. 2025Distilling Realizable Students from Unrealizable Teacherscited by 2
12 more
  1. 2026Speculative Rollback Correction for Quality-Diverse Web Agent Imitationcited by 2
  2. 2026EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learningcited by 2
  3. 2026Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agentscited by 1
  4. 2026What Drives Interactive Improvement from Feedback?cited by 1
  5. 2026CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedbackcited by 0
  6. 2026When and What to Teach: Budget-Aware Online Adaptation for Web Agentscited by 0
  7. 2026The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Taskscited by 0
  8. 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?cited by 0
  9. 2026Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agentscited by 0
  10. 2026SynAct: Synergistic Active Feedback Learning for LLM Agentscited by 0
  11. 2026Beyond Corrected Actions: Training LLM Agents on Textual Critiquescited by 0
  12. 2026Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs
  2. 2025Fine-tuning with RAG for Improving LLM Learning of New Skills
  3. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking
  4. 2026SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
  5. 2025Self-Improving LLM Agents at Test-Time
  6. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents
  7. 2025LLM Agents for Education: Advances and Applications
  8. 2026The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock