# StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows arXiv: https://arxiv.org/abs/2607.14896 · HTML: https://mlsanity.com/paper/2607.14896/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 29 of its 62 references that are in the map. - First posted on arXiv 16 Jul 2026. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 29 of 29) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023) - [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022) - [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023) - [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026) - [GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation](https://mlsanity.com/paper/2305.18997/) (2023) - [Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs](https://mlsanity.com/paper/2509.16204/) (2025) - [Integrating Large Language Models for Automated Structural Analysis](https://mlsanity.com/paper/2504.09754/) (2025) - [AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding](https://mlsanity.com/paper/2601.04819/) (2026) - [BIMgent: Towards Autonomous Building Modeling via Computer-use Agents](https://mlsanity.com/paper/2506.07217/) (2025) - [Large Language Model-Driven Code Compliance Checking in Building Information Modeling](https://mlsanity.com/paper/2506.20551/) (2025) - [Learning to simulate and design for structural engineering](https://mlsanity.com/paper/2003.09103/) (2020) - [A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysis](https://mlsanity.com/paper/2510.05414/) (2025) - [A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis](https://mlsanity.com/paper/2507.02938/) (2025) - [AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field](https://mlsanity.com/paper/2509.18776/) (2025) - [Automating Structural Engineering Workflows with Large Language Model Agents](https://mlsanity.com/paper/2510.11004/) (2025) - [AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction](https://mlsanity.com/paper/2603.29199/) (2026) - [MCP4IFC: IFC-Based Building Design Using Large Language Models](https://mlsanity.com/paper/2511.05533/) (2025) - [Automating Structural Analysis Across Multiple Software Platforms Using Large Language Models](https://mlsanity.com/paper/2604.09866/) (2026) - [Agentic Large Language Models for Automated Structural Analysis of 3D Frame Systems](https://mlsanity.com/paper/2606.06525/) (2026) ## Built on by (most cited first, 2 of 2) - [SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation](https://mlsanity.com/paper/2610.02304/) (2026) - [RocketAgent: A Long-Horizon Engineering Agent for Multidisciplinary Design of Liquid-Rocket Thrust Chambers](https://mlsanity.com/paper/2610.06044/) (2026) ## Abstract (arXiv) Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, applicable engineering checks, and a final report. Evaluations centered on question answering or script generation may therefore reward fluent outputs even when the underlying workflow is incomplete, inconsistent, or non-executable. We present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends, together with StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflows, interactive robustness, and multimodal structural-model reconstruction. Its analyzable standard and multimodal cases require both strict one-to-one structural-model matching and numerical-response agreement with frozen reference responses from the selected analysis engine; interactive cases instead require positive clarification or recovery evidence together with safe non-execution when appropriate. A trial succeeds only when every fixture-required assertion passes. Across nine text-agent configurations, generic-only execution passed the model-artifact check in 87.0% of retained outcomes but achieved only 22.0% E2E Success, whereas automatic StructureClaw reached 82.9%. Interactive and multimodal evaluations further identify semantic state consistency and executable model reconstruction as the dominant remaining bottlenecks. The code and benchmark are available at https://github.com/structureclaw/structureclaw.