# VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video

arXiv: https://arxiv.org/abs/2505.24838 · HTML: https://mlsanity.com/paper/2505.24838/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 1 references that are in the map.
- First posted on arXiv 30 May 2025.

## Built on by (most cited first, 10 of 10)

- [CADFS: A Big CAD Program Dataset and Framework for Computer-Aided Design with Large Language Models](https://mlsanity.com/paper/2605.01925/) (2026)
- [LAMP: Data-Efficient Linear Affine Weight-Space Models for Parameter-Controlled 3D Shape Generation and Extrapolation](https://mlsanity.com/paper/2510.22491/) (2025)
- [CADFit: Precise Mesh-to-CAD Program Generation with Hybrid Optimization](https://mlsanity.com/paper/2605.01171/) (2026)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing](https://mlsanity.com/paper/2604.16170/) (2026)
- [Agent-Aided Design for Dynamic CAD Models](https://mlsanity.com/paper/2604.15184/) (2026)
- [Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations](https://mlsanity.com/paper/2604.26148/) (2026)
- [Memory-Augmented Reinforcement Learning Agent for CAD Generation](https://mlsanity.com/paper/2605.19748/) (2026)
- [EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?](https://mlsanity.com/paper/2609.37686/) (2026)
- [StepCAD: Mesh-to-CAD Code Generation via LLM Policy and Geometry-Guided Search](https://mlsanity.com/paper/2610.03799/) (2026)

## Abstract (arXiv)

Computer-Aided Design (CAD) is a time-consuming and complex process, requiring precise, long-horizon user interactions with intricate 3D interfaces. While recent advances in AI-driven user interface (UI) agents show promise, most existing datasets and methods focus on short, low-complexity tasks in mobile or web applications, failing to capture the demands of professional engineering tools. In this work, we introduce VideoCAD, the first attempt to model UI interactions for precision engineering tasks. Specifically, VideoCAD is a large-scale synthetic dataset consisting of over 41K annotated video recordings of CAD operations, generated using an automated framework for collecting high-fidelity UI action data from human-made CAD designs. Compared to existing datasets, VideoCAD offers an order-of-magnitude increase in complexity for real-world engineering UI tasks, with time horizons up to 20x longer than those in other datasets. We show two important downstream applications of VideoCAD: (1) learning UI interactions from professional 3D CAD tools for precision tasks and (2) a visual question-answering (VQA) benchmark designed to evaluate multimodal large language models (LLMs) on spatial reasoning and video understanding. To learn the UI interactions, we propose VideoCADFormer, a state-of-the-art model for learning CAD interactions directly from video, which outperforms existing behavior cloning baselines. Both VideoCADFormer and the VQA benchmark derived from VideoCAD reveal key challenges in the current state of video-based UI understanding, including the need for precise action grounding, multi-modal and spatial reasoning, and long-horizon dependencies.