mlsanityData to 14 Oct 2026

NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents

arXiv 2609.33379 ↗First posted 27 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 2 more
Abstract (arXiv)

Large language model agents increasingly rely on compound programs for retrieval, tool use, reasoning, and verification, yet their failures often arise from local procedural decisions. Existing reinforcement-learning and prompt-optimization approaches typically rely on scalar rewards or repeatedly modify entire prompts, making it difficult to capture and reuse procedural improvements while preserving a frozen agent. To address this problem, We propose Natural-Language Policy Gradients (NLPG), an external policy-memory method for improving a fixed agent without changing its model parameters or program structure. NLPG diagnoses execution traces, propagates downstream feedback backward through the module graph, and converts recurring failures into route-local natural-language corrections that are aggregated into bounded policy updates for subsequent executions. Across six benchmarks covering memory, reasoning, instruction following, and evidence verification, NLPG also outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average. These results provide evidence that evaluated procedural experience can be transformed into local and interpretable policy updates, enabling continual improvement of frozen agents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 32 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

27 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  12. 2023Large Language Models as Optimizerscited by 864
14 more
  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  5. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  6. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  7. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  8. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  9. 2025Generalizing Verifiable Instruction Followingcited by 194
  10. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 180
  11. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 69
  12. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 54
  13. 2026Beyond Static Summarization: Proactive Memory Extraction for LLM Agentscited by 21
  14. 2025MemR3: Memory Retrieval via Reflective Reasoning for LLM Agentscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learning
  2. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
  3. 2024Natural Language Reinforcement Learning
  4. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  5. 2026Polaris: A G"odel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair
  6. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
  7. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
  8. 2026Toward Efficient Agents: Memory, Tool learning, and Planning