# SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

arXiv: https://arxiv.org/abs/2610.14918 · HTML: https://mlsanity.com/paper/2610.14918/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents](https://mlsanity.com/paper/2604.06132/) (2026)
- [Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence](https://mlsanity.com/paper/2604.18292/) (2026)
- [WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation](https://mlsanity.com/paper/2605.10912/) (2026)
- [Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments](https://mlsanity.com/paper/2602.01244/) (2026)
- [ClawBench: Can AI Agents Complete Everyday Online Tasks?](https://mlsanity.com/paper/2604.08523/) (2026)
- [TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents](https://mlsanity.com/paper/2602.07274/) (2026)
- [RubricBench: Aligning Model-Generated Rubrics with Human Standards](https://mlsanity.com/paper/2603.01562/) (2026)
- [Terminal-World: Scaling Terminal-Agent Environments via Agent Skills](https://mlsanity.com/paper/2605.20876/) (2026)
- [SynthAgent: Adapting Web Agents with Synthetic Supervision](https://mlsanity.com/paper/2511.06101/) (2025)
- [CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents](https://mlsanity.com/paper/2606.22883/) (2026)
- [LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks](https://mlsanity.com/paper/2604.13072/) (2026)
- [What Makes Interaction Trajectories Effective for Training Terminal Agents?](https://mlsanity.com/paper/2606.03461/) (2026)
- [ClawEnvKit: Automatic Environment Generation for Claw-Like Agents](https://mlsanity.com/paper/2604.18543/) (2026)
- [A Framework for Evaluating Agentic Skills at Scale](https://mlsanity.com/paper/2606.17819/) (2026)
- [ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents](https://mlsanity.com/paper/2605.14133/) (2026)
- [RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions](https://mlsanity.com/paper/2606.03889/) (2026)
- [RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents](https://mlsanity.com/paper/2609.17985/) (2026)

## Abstract (arXiv)

Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 (40.9% strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates (70.0% to 85.7% vs. 12.6% to 40.9%). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., Delta 19.2% for Opus 4.8 vs. Delta 5.8% for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under 3% for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.