# Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

arXiv: https://arxiv.org/abs/2412.13194 · HTML: https://mlsanity.com/paper/2412.13194/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning](https://mlsanity.com/paper/2405.10292/) (2024)
- [Autonomous Evaluation and Refinement of Digital Agents](https://mlsanity.com/paper/2404.06474/) (2024)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance](https://mlsanity.com/paper/2310.10021/) (2023)
- [OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code](https://mlsanity.com/paper/2405.15568/) (2024)
- [Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration](https://mlsanity.com/paper/2002.09253/) (2020)
- [DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents](https://mlsanity.com/paper/2410.14803/) (2024)
- [OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization](https://mlsanity.com/paper/2410.19609/) (2024)

## Built on by (most cited first, 30 of 50)

- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks](https://mlsanity.com/paper/2503.15478/) (2025)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [Reinforcement Learning for Self-Improving Agent with Skill Library](https://mlsanity.com/paper/2512.17102/) (2025)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [Self-Challenging Language Model Agents](https://mlsanity.com/paper/2506.01716/) (2025)
- [HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry](https://mlsanity.com/paper/2606.14249/) (2026)
- [Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction](https://mlsanity.com/paper/2506.07976/) (2025)
- [Go-Browse: Training Web Agents with Structured Exploration](https://mlsanity.com/paper/2506.03533/) (2025)
- [InSTA: Towards Internet-Scale Training For Agents](https://mlsanity.com/paper/2502.06776/) (2025)
- [PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction](https://mlsanity.com/paper/2510.15863/) (2025)
- [Gym-Anything: Turn any Software into an Agent Environment](https://mlsanity.com/paper/2604.06126/) (2026)
- [WebXSkill: Skill Learning for Autonomous Web Agents](https://mlsanity.com/paper/2604.13318/) (2026)
- [Scaling Synthetic Task Generation for Agents via Exploration](https://mlsanity.com/paper/2509.25047/) (2025)
- [ProgRM: Build Better GUI Agents with Progress Rewards](https://mlsanity.com/paper/2505.18121/) (2025)
- [WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis](https://mlsanity.com/paper/2507.04370/) (2025)
- [WebGuard: Building a Generalizable Guardrail for Web Agents](https://mlsanity.com/paper/2507.14293/) (2025)
- [Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning](https://mlsanity.com/paper/2504.21561/) (2025)
- [WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks](https://mlsanity.com/paper/2601.02439/) (2026)
- [LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries](https://mlsanity.com/paper/2508.15760/) (2025)
- [Evaluating Long-Context Reasoning in LLM-Based WebAgents](https://mlsanity.com/paper/2512.04307/) (2025)
- [Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback](https://mlsanity.com/paper/2506.04287/) (2025)
- [Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent](https://mlsanity.com/paper/2508.04037/) (2025)
- [Safe and Scalable Web Agent Learning via Recreated Websites](https://mlsanity.com/paper/2603.10505/) (2026)
- [Orchard: An Open-Source Agentic Modeling Framework](https://mlsanity.com/paper/2605.15040/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [OpenForgeRL: Train Harness-native Agents in Any Environment](https://mlsanity.com/paper/2607.21557/) (2026)
- [Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution](https://mlsanity.com/paper/2505.17673/) (2025)
- [WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents](https://mlsanity.com/paper/2603.05044/) (2026)

## Abstract (arXiv)

The vision of a broadly capable and goal-directed agent, such as an Internet-browsing agent in the digital world and a household humanoid in the physical world, has rapidly advanced, thanks to the generalization capability of foundation models. Such a generalist agent needs to have a large and diverse skill repertoire, such as finding directions between two travel locations and buying specific items from the Internet. If each skill needs to be specified manually through a fixed set of human-annotated instructions, the agent's skill repertoire will necessarily be limited due to the quantity and diversity of human-annotated instructions. In this work, we address this challenge by proposing Proposer-Agent-Evaluator, an effective learning system that enables foundation model agents to autonomously discover and practice skills in the wild. At the heart of PAE is a context-aware task proposer that autonomously proposes tasks for the agent to practice with context information of the environment such as user demos or even just the name of the website itself for Internet-browsing agents. Then, the agent policy attempts those tasks with thoughts and actual grounded operations in the real world with resulting trajectories evaluated by an autonomous VLM-based success evaluator. The success evaluation serves as the reward signal for the agent to refine its policies through RL. We validate PAE on challenging vision-based web navigation, using both real-world and self-hosted websites from WebVoyager and WebArena.To the best of our knowledge, this work represents the first effective learning system to apply autonomous task proposal with RL for agents that generalizes real-world human-annotated benchmarks with SOTA performances. Our open-source checkpoints and code can be found in https://yanqval.github.io/PAE/