mlsanityData to 14 Oct 2026

ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

arXiv 2411.15004 ↗First posted 22 Nov 2024cs.CL cs.AIcited by 39

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 49 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

22 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent Workflow Memory11 Sep
  2. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?12 Mar
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  9. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  10. 2024Tree Search for Language Model Agentscited by 151
  11. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  12. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
7 more
  1. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
  2. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  3. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  4. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  5. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  6. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  7. 2022Understanding HTML with Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 28; 2026: 10.

39 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  5. 2025Agent Learning via Early Experiencecited by 85
  6. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  7. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  8. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 39
  9. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  10. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 33
  11. 2025InSTA: Towards Internet-Scale Training For Agentscited by 32
  12. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
27 more
  1. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  2. 2025CAT: Content-Adaptive Image Tokenizationcited by 21
  3. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  4. 2025Context manipulation attacks : Web agents are susceptible to corrupted memorycited by 12
  5. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 12
  6. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 12
  7. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  8. 2025WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environmentcited by 10
  9. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  10. 2026OpAgent: Operator Agent for Web Navigationcited by 8
  11. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  12. 2025Reflection-Based Memory For Web navigation Agentscited by 5
  13. 2025From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Productioncited by 5
  14. 2026Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectoriescited by 4
  15. 2025Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agentscited by 3
  16. 2026Benchmarking Web Agent Safety under E-commerce Deceptive Interfacescited by 3
  17. 2025FormGym: Doing Paperwork with Agentscited by 2
  18. 2025Learning Efficient Guardrails for Compliancecited by 2
  19. 2025RECODE: Reasoning Through Code Generation for Visual Question Answeringcited by 2
  20. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  21. 2026ProCUA-SFT Technical Reportcited by 2
  22. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  23. 2026WebChallenger: A Reliable and Efficient Generalist Web Agentcited by 1
  24. 2026Software Engineering for and with GUI Agentcited by 1
  25. 2026CORE: Code-based Inverse Self-Training Framework with Graph Expansion for Virtual Agentscited by 0
  26. 2026SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agentscited by 0
  27. 2026The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving LLM-based Autonomous Web Agents with Filtering
  2. 2023Mind2Web: Towards a Generalist Agent for the Web
  3. 2024Large Language Models Can Self-Improve At Web Agent Tasks
  4. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
  5. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
  6. 2026LLM Agents Factory: Retrieval of Domain-Specific LLM Agents
  7. 2023Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
  8. 2024The BrowserGym Ecosystem for Web Agent Research