mlsanityData to 7 Oct 2026

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

arXiv 2609.37686 ↗First posted 29 Sep 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constraints and dependencies preserved across software and design stages. We present EngiWorld, the first benchmark structured around the complete design loop: 1,301 expert-curated tasks spanning 6 engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces and 6 task types ranging from software-selection to open-ended tasks. We further introduce an artifact-centric evaluation methodology built on a unified domain-verifier suite, which programmatically checks the geometric validity, physical feasibility, and rule compliance of final and intermediate artifacts, and scores quantitative design tasks continuously by specification attainment rather than binary success. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed. EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

27 of 39 references.

References · 27

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  6. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  7. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  8. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 266
  9. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  10. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  11. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  12. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 143
15 more
  1. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
  2. 2023RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solutioncited by 69
  3. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  4. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  5. 2024MetaOpenFOAM: an LLM-based multi-agent framework for CFDcited by 39
  6. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  7. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  8. 2025FEABench: Evaluating Language Models on Multiphysics Reasoning Abilitycited by 21
  9. 2025Foam-Agent 2.0: An End-to-End Composable Multi-Agent Framework for Automating CFD Simulation in OpenFOAMcited by 15
  10. 2025VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Videocited by 9
  11. 2026Text-to-CAD Evaluation with CADTestscited by 9
  12. 2026TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learningcited by 6
  13. 2025Using GUI Agent for Electronic Design Automationcited by 5
  14. 2026CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulationcited by 3
  15. 2026BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modelingcited by 1

Built on by

Not cited yet.