mlsanityData to 14 Oct 2026

MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models

arXiv 2510.04363 ↗First posted 5 Oct 2025cs.SE cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce MacroBench, a code-first benchmark that evaluates whether LLMs can synthesize reusable browser-automation programs (macros) from natural-language goals by reading HTML/DOM and emitting Selenium. MacroBench instantiates seven self-hosted sites covering 681 tasks across interaction complexity and targeting difficulty. Our end-to-end protocol validates generated code via static checks, sandboxed execution, and outcome verification (DOM assertions, database snapshots), and includes a safety suite for scraping, spam/abuse, and credential/privacy prompts. Across 2,636 model-task runs, we observe stratified success: GPT-4o-mini (96.8%), GPT-4o (95.3%), Gemini (89.0%), DeepSeek (83.4%). Models handle simple tasks reliably (91.7%) but fail on complex workflows (0.0%), and none meet production-quality coding practices despite functional completion. We release our complete benchmark pipeline, evaluation framework, and experimental results at https://github.com/hyunjun1121/MacroBench to enable reproducible assessment of macro synthesis for web automation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

25 of 30 references.

References · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
13 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2022PAL: Program-aided Language Modelscited by 652
  4. 2025Humanity's Last Examcited by 491
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  6. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  7. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  8. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  9. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  10. 2019DOM-Q-NET: Grounded RL on Structured Languagecited by 25
  11. 2023Automatic Macro Mining from Interaction Traces at Scalecited by 12
  12. 2025ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreakscited by 3
  13. 2025X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templatescited by 1

Built on by

Not cited yet.