mlsanityData to 7 Oct 2026

Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

arXiv 2308.02151 ↗First posted 4 Aug 2023cs.CL cs.AIcited by 110

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment. This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 31 references that are in the map. First posted on arXiv 4 Aug 2023.

Builds on

22 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
9 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  8. 2023Learning Performance-Improving Code Editscited by 103
  9. 2018Counting to Explore and Generalize in Text-based Gamescited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 36; 2025: 31; 2026: 30.

110 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  3. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  4. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  5. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  9. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  10. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2025A Survey of Context Engineering for Large Language Modelscited by 120
38 more
  1. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  3. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  4. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  6. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  7. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  8. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  9. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  10. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  11. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  12. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  13. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  14. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  15. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  16. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  17. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  18. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  19. 2024Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Modelscited by 35
  20. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  21. 2024Planning Like Human: A Dual-process Framework for Dialogue Planningcited by 32
  22. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30
  23. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29
  24. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  25. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  26. 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Modelscited by 25
  27. 2024MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistantscited by 24
  28. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  29. 2026Aligning Language Models from User Interactionscited by 21
  30. 2024Empowering Large Language Model Agents through Action Learningcited by 20
  31. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  32. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  33. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 17
  34. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Maskingcited by 16
  35. 2025Coarse-to-Fine Grounded Memory for LLM Agent Planningcited by 16
  36. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  37. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  38. 2024Driving Style Alignment for LLM-powered Driver Agentcited by 15

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents
  2. 2024MetaReflection: Learning Instructions for Language Agents using Past Reflections
  3. 2025Retrospex: Language Agent Meets Offline Reinforcement Learning Critic
  4. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  5. 2025Prompt reinforcing for long-term planning of large language models
  6. 2024RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agents
  7. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
  8. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuning