mlsanityData to 14 Oct 2026

Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration

arXiv 2604.18131 ↗First posted 20 Apr 2026cs.AIcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Most agents today ``self-evolve'' by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent's self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 48 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

42 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Kimi K2: Open Agentic Intelligence28 Jul
  2. 2025Self-Challenging Language Model Agents2 Jun
  3. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents29 May
  4. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments18 Jan
  5. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Space8 Oct
  6. 2023ExpeL: LLM Agents Are Experiential Learners20 Aug

Other references · 36

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  5. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  10. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  11. 2024Titans: Learning to Memorize at Test Timecited by 328
  12. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
24 more
  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  3. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  4. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
  5. 2025Agent Learning via Early Experiencecited by 85
  6. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  7. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  8. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  9. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  10. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
  11. 2025Test-Time Learning for Large Language Modelscited by 50
  12. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  13. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  14. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  15. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 37
  16. 2025Self-Supervised Prompt Optimizationcited by 37
  17. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 22
  18. 2025LLM-AutoDiff: Auto-Differentiate Any LLM Workflowcited by 21
  19. 2025AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learningcited by 14
  20. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selectioncited by 13
  21. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 13
  22. 2025Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planningcited by 12
  23. 2025WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Modelscited by 10
  24. 2026Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memoriescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 12
  2. 2026What is Missing from AI Post-Training AI: An Empirical Analysiscited by 2
  3. 2026Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EvoBelief: Falsifiable World Models for Self-Improving LLM Agents
  2. 2026OpenSkill: Open-World Self-Evolution for LLM Agents
  3. 2025Towards Agentic Self-Learning LLMs in Search Environment
  4. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
  5. 2026Aspire: Can Models Self-Evolve from Vague Goals?
  6. 2025AgentEvolver: Towards Efficient Self-Evolving Agent System
  7. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
  8. 2026MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs