mlsanityData to 14 Oct 2026

ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm

arXiv 2606.13239 ↗First posted 11 Jun 2026cs.SE cs.AI cs.CLcited by 2

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Existing computer-use agents remain fundamentally limited in professional software manipulation: GUI-based agents suffer from fragile visual grounding and long-horizon error accumulation, while API-basedapproaches struggle with heterogeneous protocols and inaccessible commercial interfaces. In this work,we identify the Component Object Model (COM) as a unified executable abstraction, proposing COM-as-Action: a new paradigm that reframes professional software interaction as deterministic program synthesisrather than sequential visual control. To validate this paradigm in the most demanding environments, weintroduce ComCADBench, the first benchmark for agents operating real industrial CAD software. Ourexperiments reveal a substantial paradigm gap: frontier proprietary models achieve near-zero successunder GUI-based interaction, whereas COM-based execution yields substantial immediate gains. Tobridge the remaining gap between syntactic correctness and geometric accuracy, we develop ComActor, aself-correcting agent trained through a progressive three-stage framework, alongside ComForge, a scalableplatform for large-scale training in Windows containers. Extensive experiments show that ComActorachieves state-of-the-art performance on ComCADBench, with strong resilience in long-horizon taskswhere baselines collapse, and generalizes to external CAD benchmark.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 40 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

30 of 40 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  3. 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
  4. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  7. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  8. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2025OpenAI GPT-5 System Cardcited by 989
  7. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  10. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  12. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
10 more
  1. 2024Beyond Browsing: API-Based Web Agentscited by 78
  2. 2020SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Designcited by 72
  3. 2025CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generationcited by 67
  4. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 66
  5. 2025CAD-Coder: Text-to-CAD Generation with Chain-of-Thought and Geometric Rewardcited by 56
  6. 2024FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Modelscited by 34
  7. 2024TransCAD: A Hierarchical Transformer for CAD Sequence Inference from Point Cloudscited by 29
  8. 2025CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Designcited by 25
  9. 2025CAD-Judge: Toward Efficient Morphological Grading and Verification for Text-to-CAD Generationcited by 10
  10. 2025UFO3: Weaving the Digital Agent Galaxycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Step-level Optimization for Efficient Computer-use Agents
  2. 2026Computer-Using World Model
  3. 2025Programming with Pixels: Can Computer-Use Agents do Software Engineering?
  4. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  5. 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
  6. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
  7. 2025XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level
  8. 2026CLI-Anything: Towards Agent-Native Computer Use