mlsanityData to 14 Oct 2026

Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?

arXiv 2609.35897 ↗First posted 27 Sep 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The pursuit of recursive self-improvement (RSI) toward general intelligence is divided between macro-level language model scaling and the interaction-driven principles of "Era of Experience". Yet, any self-improving architecture ultimately rests upon its underlying optimization engine: if general intelligence requires learning from grounded interaction, the reinforcement learning (RL) update rule itself must be capable of cumulative adaptation. While algorithm self-discovery has produced Disco103 that surpassed PPO to achieve SOTA benchmark performance -- its internal update machinery remains an uninspected black box. We present the first causal mechanistic audit of a self-discovered RL rule, structured directly around the five pillars of the Era of Experience: extended horizon, grounded reward scales, continuing streams, within-lifetime change, and exploration depth. By surgically pinning, freezing, and transplanting recurrent states while holding meta-parameters fixed, we test when learning history acts as an asset or a burden. Three findings organize the audit: (1) Recurrent history actively expands usable reward scales, sustaining a six-decade window versus three under zero-pinning. (2) Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates this burden. (3) Under environmental change, controlling replay retention reverses the apparent adaptation advantage over DQN, demonstrating that external data turnover can confound internal plasticity. Validated through capability thresholds and ported to a second rule (OPEN), this work grounds macro-RSI ambitions in micro-level learning dynamics, establishing a foundational audit standard for next-generation, self-evolving RL algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 3 of its 28 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

3 of 28 references.

References · 3

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents
  2. 2026COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvement
  3. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learning
  4. 2026Rethinking Experience Utilization in Self-Evolving Language Model Agents
  5. 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
  6. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
  7. 2024When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions
  8. 2026SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents