mlsanityData to 14 Oct 2026

Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

arXiv 2402.17574 ↗First posted 27 Feb 2024cs.AI cs.CLcited by 94

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) exhibit robust problem-solving capabilities for diverse tasks. However, most LLM-based agents are designed as specific task solvers with sophisticated prompt engineering, rather than agents capable of learning and evolving through interactions. These task solvers necessitate manually crafted prompts to inform task rules and regulate LLM behaviors, inherently incapacitating to address complex dynamic scenarios e.g., large interactive games. In light of this, we propose Agent-Pro: an LLM-based Agent with Policy-level Reflection and Optimization that can learn a wealth of expertise from interactive experiences and progressively elevate its behavioral policy. Specifically, it involves a dynamic belief generation and reflection process for policy evolution. Rather than action-level reflection, Agent-Pro iteratively reflects on past trajectories and beliefs, fine-tuning its irrational beliefs for a better policy. Moreover, a depth-first search is employed for policy optimization, ensuring continual enhancement in policy payoffs. Agent-Pro is evaluated across two games: Blackjack and Texas Hold'em, outperforming vanilla LLM and specialized models. Our results show Agent-Pro can learn and evolve in complex and dynamic scenes, which also benefits numerous LLM-based applications.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 48 of its 67 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

48 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf9 Sep

Other references · 47

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
35 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2022Large Language Models Can Self-Improvecited by 664
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  8. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  10. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  11. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  12. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  13. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  14. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
  15. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  16. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  17. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  18. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  19. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  20. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 92
  21. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  22. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  23. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  24. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 68
  25. 2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 66
  26. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54
  27. 2023Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaborationcited by 52
  28. 2019RLCard: A Toolkit for Reinforcement Learning in Card Gamescited by 35
  29. 2022Large Language Models can Implement Policy Iterationcited by 31
  30. 2023Introspective Tips: Large Language Model for In-Context Decision Makingcited by 28
  31. 2024Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolutioncited by 28
  32. 2022Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problemcited by 13
  33. 2022A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigationcited by 6
  34. 2021Learning to Navigate in a VUCA Environment: Hierarchical Multi-expert Approachcited by 5
  35. 2023An Expression Tree Decoding Strategy for Mathematical Equation Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 42; 2026: 32.

94 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  3. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 113
  8. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  9. 2024Symbolic Learning Enables Self-Evolving Agentscited by 106
  10. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
  11. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  12. 2025SWE-Exp: Experience-Driven Software Issue Resolutioncited by 65
38 more
  1. 2024AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learningcited by 61
  2. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  3. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  4. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  5. 2024On the Structural Memory of LLM Agentscited by 37
  6. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  7. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 29
  8. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  9. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  10. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  11. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  12. 2024REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimizationcited by 20
  13. 2025SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligencecited by 18
  14. 2025Coarse-to-Fine Grounded Memory for LLM Agent Planningcited by 18
  15. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 17
  16. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  17. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  18. 2026SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scalecited by 16
  19. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  20. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 12
  21. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  22. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspectivecited by 10
  23. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10
  24. 2025Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaborationcited by 10
  25. 2025Alita-G: Self-Evolving Generative Agent for Agent Generationcited by 10
  26. 2026GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)cited by 10
  27. 2025SE-VLN: A Self-Evolving Vision-Language Navigation Framework Based on Multimodal Large Language Modelscited by 9
  28. 2026Combee: Scaling Prompt Learning for Self-Improving Language Model Agentscited by 9
  29. 2024EscapeBench: Towards Advancing Creative Intelligence of Language Model Agentscited by 6
  30. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 6
  31. 2026FederatedSkill: Federated Learning for Agentic Skill Evolutioncited by 6
  32. 2024Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoningcited by 5
  33. 2025Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalizationcited by 5
  34. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  35. 2025EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVercited by 5
  36. 2026SkillCommit: Evolving Agent Skills through Behaviorally Validated Scope Expansioncited by 5
  37. 2025Regression and Forecasting of U.S. Stock Returns Based on LSTMcited by 4
  38. 2025Reflection of Episodes: Learning to Play Game from Expert and Self Experiencescited by 4

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  2. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
  3. 2025Policy-Conditioned Policies for Multi-Agent Task Solving
  4. 2026MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolution
  5. 2025PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training
  6. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  7. 2025Agents of Change: Self-Evolving LLM Agents for Strategic Planning
  8. 2025Dynamic Strategy Adaptation in Multi-Agent Environments with Large Language Models