# Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.17591 · HTML: https://mlsanity.com/paper/2606.17591/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 15 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 13 of 13)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Hyperagents](https://mlsanity.com/paper/2603.19461/) (2026)
- [Training-Free Group Relative Policy Optimization](https://mlsanity.com/paper/2510.08191/) (2025)
- [FLEX: Continuous Agent Evolution via Forward Learning from Experience](https://mlsanity.com/paper/2511.06449/) (2025)
- [Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects](https://mlsanity.com/paper/2512.12818/) (2025)
- [Experiential Reflective Learning for Self-Improving LLM Agents](https://mlsanity.com/paper/2603.24639/) (2026)
- [Graph-Native Cognitive Memory for AI Agents: Formal Belief Revision Semantics for Versioned Memory Architectures](https://mlsanity.com/paper/2603.17244/) (2026)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [Hindsight Preference Optimization for Financial Time Series Advisory](https://mlsanity.com/paper/2604.23988/) (2026)

## Built on by (most cited first, 3 of 3)

- [The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents](https://mlsanity.com/paper/2607.07436/) (2026)
- [ERRAND: Budgeted Maintenance of Agent Memory](https://mlsanity.com/paper/2609.29545/) (2026)
- [Towards Reversible Forgetting: Managing Obsolete Knowledge in Continual Enterprise AI Agents](https://mlsanity.com/paper/2608.18177/) (2026)

## Abstract (arXiv)

Training-free verbal reinforcement learning enables LLM agents to learn from world feedback -- objective signals such as dynamic task outcomes, market returns, or demand forecasts -- by extracting verbal rules from experience and injecting them as context, updating the agent's behavior without parameter changes. However, in non-stationary environments these agents face a retention-forgetting dilemma: retaining stale insights causes negative transfer, while discarding them causes catastrophic forgetting when conditions recur. We identify four requirements for navigating this dilemma -- outcome-driven evaluation, persistent structured evidence, non-monotonic knowledge lifecycle, and compositional governance -- and show that existing methods invest heavily in experience extraction while underinvesting in insight governance. We propose a three-layer architecture -- rules, evidence, and skills -- connected by a feedback-driven curation loop that closes the governance gap. Rules capture distilled experience from world outcomes; evidence logs track each rule's reliability across episodes; skills govern which rules to apply, how to resolve conflicts, and when to abstain. On financial forecasting as a case study, where world feedback is naturally abundant, noisy, and non-stationary, we show that the same accumulated experience either degrades performance below the zero-shot baseline or dramatically improves accuracy and risk-adjusted returns, depending on whether the curation loop is present.