# mrCAD: Multimodal Refinement of Computer-aided Designs

arXiv: https://arxiv.org/abs/2504.20294 · HTML: https://mlsanity.com/paper/2504.20294/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design](https://mlsanity.com/paper/2007.08506/) (2020)
- [AutoPresent: Designing Structured Visuals from Scratch](https://mlsanity.com/paper/2501.00912/) (2025)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions](https://mlsanity.com/paper/2312.12450/) (2023)
- [Generic 3D Diffusion Adapter Using Controlled Multi-View Editing](https://mlsanity.com/paper/2403.12032/) (2024)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)

## Built on by (most cited first, 1 of 1)

- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)

## Abstract (arXiv)

A key feature of human collaboration is the ability to iteratively refine the concepts we have communicated. In contrast, while generative AI excels at the textitgeneration of content, it often struggles to make specific language-guided textitmodifications of its prior outputs. To bridge the gap between how humans and machines perform edits, we present mrCAD, a dataset of multimodal instructions in a communication game. In each game, players created computer aided designs (CADs) and refined them over several rounds to match specific target designs. Only one player, the Designer, could see the target, and they must instruct the other player, the Maker, using text, drawing, or a combination of modalities. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. We analyze the dataset and find that generation and refinement instructions differ in their composition of drawing and text. Using the mrCAD task as a benchmark, we find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay a foundation for analyzing and modeling a multimodal language of refinement that is not represented in previous datasets.