# Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback

arXiv: https://arxiv.org/abs/2506.04287 · HTML: https://mlsanity.com/paper/2506.04287/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 48 of its 51 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 30 of 48)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Tree Search for Language Model Agents](https://mlsanity.com/paper/2407.01476/) (2024)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)

## Built on by (most cited first, 12 of 12)

- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History](https://mlsanity.com/paper/2606.08671/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [Online Skill Learning for Web Agents via State-Grounded Dynamic Retrieval](https://mlsanity.com/paper/2606.04391/) (2026)
- [Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution](https://mlsanity.com/paper/2605.08887/) (2026)
- [Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning](https://mlsanity.com/paper/2509.22008/) (2025)
- [Toward Latent Language Model Skills Steering and Optimization: An Empirical Study](https://mlsanity.com/paper/2608.29459/) (2026)
- [Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security](https://mlsanity.com/paper/2608.29596/) (2026)
- [SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles](https://mlsanity.com/paper/2610.09832/) (2026)
- [Davinci-Tool: Scaling Agentic Tool-Use Supervision with Verifiable Executable Environments](https://mlsanity.com/paper/2610.16867/) (2026)

## Abstract (arXiv)

Training large language model (LLM) agents to acquire necessary skills and perform diverse tasks within an environment is gaining interest as a means to enable open-endedness. However, creating the training dataset for their skill acquisition faces several challenges. Manual trajectory collection requires significant human effort. Another approach, where LLMs directly propose tasks to learn, is often invalid, as the LLMs lack knowledge of which tasks are actually feasible. Moreover, the generated data may not provide a meaningful learning signal, as agents often already perform well on the proposed tasks. To address this, we propose a novel automatic skill discovery framework EXIF for LLM-powered agents, designed to improve the feasibility of generated target behaviors while accounting for the agents' capabilities. Our method adopts an exploration-first strategy by employing an exploration agent (Alice) to train the target agent (Bob) to learn essential skills in the environment. Specifically, Alice first interacts with the environment to retrospectively generate a feasible, environment-grounded skill dataset, which is then used to train Bob. Crucially, we incorporate an iterative feedback loop, where Alice evaluates Bob's performance to identify areas for improvement. This feedback then guides Alice's next round of exploration, forming a closed-loop data generation process. Experiments on Webshop and Crafter demonstrate EXIF's ability to effectively discover meaningful skills and iteratively expand the capabilities of the trained agent without any human intervention, achieving substantial performance improvements. Interestingly, we observe that setting Alice to the same model as Bob also notably improves performance, demonstrating EXIF's potential for building a self-evolving system.