mlsanityData to 7 Oct 2026

Agent Learning via Early Experience

arXiv 2510.08558 ↗First posted 9 Oct 2025cs.AI cs.CL cs.IRcited by 79

Not on the map. Why?

Cited by 46 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks. However, training agents from experience data with reinforcement learning remains difficult in many environments, which either lack verifiable rewards (e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn tool use). As a result, most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly. This limitation stems from the nature of expert demonstrations: they capture only a narrow range of scenarios, and expose the agent to limited environment diversity. We address this limitation with a middle-ground paradigm we call early experience: interaction data generated by the agent's own actions, where the resulting future states serve as supervision without reward signals. Within this paradigm, we study two strategies of using such data: (1) implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making. Evaluation across eight diverse environments and multiple model families shows that our approaches consistently improve effectiveness and out-of-domain generalization, highlighting the value of early experience. Moreover, in environments with verifiable rewards, our results provide promising signals that early experience offers a strong foundation for subsequent reinforcement learning, making it a practical bridge between imitation learning and fully experience-driven agents.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 69 of its 95 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

69 of 95 references.

References · 69

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
38 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2020Mastering Atari with Discrete World Modelscited by 990
  13. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  14. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  15. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  16. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  17. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  18. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  19. 2022Large Language Models Can Self-Improvecited by 656
  20. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  21. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  22. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  23. 2023Cognitive Architectures for Language Agentscited by 422
  24. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  25. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  26. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  27. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  28. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  29. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  30. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  31. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  32. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  33. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  34. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  35. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  36. 2024Tree Search for Language Model Agentscited by 148
  37. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  38. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 70.

79 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2026Reinforcement Learning via Self-Distillationcited by 285
  3. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  4. 2026Qwen3-Coder-Next Technical Reportcited by 60
  5. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  6. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  7. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  8. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  9. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  10. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  11. 2026TimeART: Towards Agentic Time Series Reasoning via Tool-Augmentationcited by 25
  12. 2026Online Experiential Learning for Language Modelscited by 24
38 more
  1. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
  2. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 18
  3. 2026SWE-Universe: Scale Real-World Verifiable Environments to Millionscited by 16
  4. 2026Learning to Continually Learn via Meta-learning Agentic Memory Designscited by 16
  5. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
  6. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  7. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  8. 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 12
  9. 2025Agentic Learner with Grow-and-Refine Multimodal Semantic Memorycited by 11
  10. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  11. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  12. 2026QUEST: Training Frontier Deep Research Agents with Fully Synthetic Taskscited by 8
  13. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 8
  14. 2026Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Modelscited by 7
  15. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agentscited by 6
  16. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experiencecited by 6
  17. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  18. 2026Agent-Omit: Adaptive Context Omission for Efficient LLM Agentscited by 5
  19. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  20. 2026From History to State: Constant-Context Skill Learning for LLM Agentscited by 5
  21. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 5
  22. 2026Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Taskscited by 4
  23. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 4
  24. 2026TARSE: Test-Time Adaptation via Retrieval of Skills and Experience for Reasoning Agentscited by 4
  25. 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamicscited by 4
  26. 2025VisCoder2: Building Multi-Language Visualization Coding Agentscited by 3
  27. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  28. 2025Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Draggingcited by 3
  29. 2026GLOVE: Global Verifier for LLM Memory-Environment Realignmentcited by 3
  30. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 3
  31. 2026The Tool Illusion: Rethinking Tool Use in Web Agentscited by 3
  32. 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Explorationcited by 3
  33. 2026Learning Agent Routing From Early Experiencecited by 3
  34. 2026GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memorycited by 3
  35. 2026Scaling GUI Agents with Visual State Transitionscited by 3
  36. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  37. 2025Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interactioncited by 2
  38. 2026Agentic Critical Trainingcited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Experience Utilization in Self-Evolving Language Model Agents
  2. 2025Contextual Experience Replay for Self-Improvement of Language Agents
  3. 2026Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
  4. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback
  5. 2026PRACTICE: From Experience to Expertise in Self-Evolving Embodied Agents
  6. 2025Don't Just Fine-tune the Agent, Tune the Environment
  7. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?
  8. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective