# BAGEL: Bootstrapping Agents by Guiding Exploration with Language

arXiv: https://arxiv.org/abs/2403.08140 · HTML: https://mlsanity.com/paper/2403.08140/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 30 of 35)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [Keep CALM and Explore: Language Models for Action Generation in Text-based Games](https://mlsanity.com/paper/2010.02903/) (2020)
- [A data-driven approach for learning to control computers](https://mlsanity.com/paper/2202.08137/) (2022)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces](https://mlsanity.com/paper/2306.00245/) (2023)

## Built on by (most cited first, 30 of 34)

- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model](https://mlsanity.com/paper/2504.21024/) (2025)
- [NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild](https://mlsanity.com/paper/2410.02907/) (2024)
- [Self-Challenging Language Model Agents](https://mlsanity.com/paper/2506.01716/) (2025)
- [Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale](https://mlsanity.com/paper/2409.15637/) (2024)
- [ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data](https://mlsanity.com/paper/2411.15004/) (2024)
- [A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions](https://mlsanity.com/paper/2501.16150/) (2025)
- [Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction](https://mlsanity.com/paper/2506.07976/) (2025)
- [Go-Browse: Training Web Agents with Structured Exploration](https://mlsanity.com/paper/2506.03533/) (2025)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [Gym-Anything: Turn any Software into an Agent Environment](https://mlsanity.com/paper/2604.06126/) (2026)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Build the web for agents, not agents for the web](https://mlsanity.com/paper/2506.10953/) (2025)
- [Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting](https://mlsanity.com/paper/2510.10304/) (2025)
- [Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback](https://mlsanity.com/paper/2506.04287/) (2025)
- [CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL](https://mlsanity.com/paper/2512.01311/) (2025)
- [Watch and Learn: Learning to Use Computers from Online Videos](https://mlsanity.com/paper/2510.04673/) (2025)
- [HATS: Hardness-Aware Trajectory Synthesis for GUI Agents](https://mlsanity.com/paper/2603.12138/) (2026)
- [Agentic Test-Time Scaling for WebAgents](https://mlsanity.com/paper/2602.12276/) (2026)
- [Improving Large Language Model Planning with Action Sequence Similarity](https://mlsanity.com/paper/2505.01009/) (2025)
- [Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations](https://mlsanity.com/paper/2607.04235/) (2026)
- [Code-enabled language models can outperform reasoning models on diverse tasks](https://mlsanity.com/paper/2510.20909/) (2025)
- [Structured Distillation of Web Agent Capabilities Enables Generalization](https://mlsanity.com/paper/2604.07776/) (2026)
- [EnterpriseLab: A Full-Stack Platform for developing and deploying agents in Enterprises](https://mlsanity.com/paper/2603.21630/) (2026)

## Abstract (arXiv)

Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generalize to new environments without human demonstrations. This work presents BAGEL, a method for bootstrapping LM agents without human supervision. BAGEL converts a seed set of randomly explored trajectories or synthetic instructions, into demonstrations, via round-trips between two noisy LM components: an LM labeler which converts a trajectory into a synthetic instruction, and a zero-shot LM agent which maps the synthetic instruction into a refined trajectory. By performing these round-trips iteratively, BAGEL quickly converts the initial distribution of trajectories towards those that are well-described by natural language. We use BAGEL demonstrations to adapt a zero shot LM agent at test time via in-context learning over retrieved demonstrations, and find improvements of over 2-13% absolute on ToolQA and MiniWob++, with up to 13x reduction in execution failures.